Dear L2 Pulsar Upgrade Interested Parties: The L2 Pulsar readiness review committee commends the L2 Pulsar group for their hard work and committment in bringing the L2 Pulsar project up to the integration and commissioning phase. We think that the Pulsar group did an excellent job in informing the committee on the current status of the L2 Pulsar system at the 9 Feb 2005 L2 Pulsar second readiness review. From this review, we conclude that operational integration in March 2005 is attainable. As stated in the committee's first readiness review report, attaining the March 2005 integration date is also necessary to insure that the L2 Pulsar system is smoothly operational before the numerous trigger upgrades planned for the August 2005 shutdown are underway. Therefore, we urge and fully support the L2 Pulsar group in their efforts to attain this goal. Most of the issues raised in the first readiness review have been addressed by the Pulsar group in a timely fashion, and we again thank the Pulsar group for their dedication and hard work. The committee believes that the Pulsar group is on track with the project timeline from the first readiness review, and finds no significant problems. We present requests and recommendations that we believe are necessary and will assist the Pulsar group in making the L2 Pulsar system operational. The requests and recommendations of the L2 Pulsar readiness review committee are grouped in two time-frame categories: 1) Tests and debugging in preparation for operations: Before 1 Mar 2005 2) Operational integration : After 1 Mar 2005. The date should be considered as a target date. The two categories are detailed in the list below. The category 2) period is physics quality running, ie, the online goodrun status must be "good". If the category 1) issues are addressed and resolved prior to 25 Feb 2005, the Pulsar group is requested to present a progress report on their actions in category 1) and their preparations for category 2) running at the 25 Feb 2005 Trigger Hardware Meeting slot: Pump Room, 1000-1130 hrs. However, the actual date for the progress report meeting and category 2) running is dependent on the successful completion of the items in category 1). At this point, with the blessing of the committee and CDF Operations group, the category 2) operations will begin. If at the end of the category 2) period there are no signficant problems, we consider the L2 Pulsar Upgrade project to be complete. Any trouble would reset the clock. To facilitate the switchover from the IIa Alpha trigger algorithms to the IIb Pulsar algorithms, the TDWG heads will limit trigger algorithm development. However, it is important to verify that trigger algorithm development can proceed with the Pulsar. Any trigger table that is built during this period will be build with both Alpha and Pulsar versions. The L2 Pulsar group should be ready for these limited changes. The committee recommends that the legacy Alpha system shall remain powered and ready for use for a minimum of one month after the completion of the category 2) period. New trigger tables during this period should be built for both systems. After this one month period until the begining of the 2005 Shutdown, the Legacy hardware shall remain in place but not necessarily powered. Trigger tables will continue to be built for the legacy system as long as this does not require writing new code for the Alpha. During the 2005 Shutdown, the legacy hardware may be removed and support of the legacy system in online software and monitoring may end. The committee is still concerned about an issue raised in the first L2 Pulsar readiness review: MOUs that address long-term operations and institutional support for the components of the Pulsar system. The committee recommends the Pulsar group to firm up and produce MOUs or e-mail commitments on MOUs for all parts of the hardware, firmware, and software. We strongly urge that this process be finalized, or in the final states by the end of the category 2) Operations integration period. The committee requests that the Pulsar group provide written documentation about the MOUs. The committee considers the MOUs as a necessary part of getting the L2 Pulsar operational and in its blessing. The committee suggests that the L2 Pulsar group secure operations committments from institutions currently involved in the development and commissioning, as has been done with the University of Pennsylvania. Any impediments in establishing these MOUs should be brought to the attention of the Spokespersons and the Operations Department. The committee has the following recommendations to the Pulsar group pertaining to upgrades: a) As the RECES splitters are working, we recommend that they should not be touched until the upcoming shutdown. During the shutdown, an attempt should be made to determine why the error rate has decreased significantly with the splitter for the IIA system. b) We recommend that the Pulsar group devote most of their efforts to the FILAR board rather than attempting to rewrite firmware for the existing PCI card. Best wishes, The L2 Pulsar Readiness Review Committee: Bill Badgett, Roberto Carosi, Camille Ginsburg, Jonathan Lewis, Pat Lukens, Steve Nahn, Jim Patrick, Kevin Pitts, Rob Roser, Willis Sakumoto, David Saltzberg, Michael Schmidt, Kirsten Tollefson, and Tom Wright The Time-frame categories 1) and 2) ----------------------------------- 1) Tests and debugging in preparation for operations a) Prepare a clear, priority-ordered task lists with assigned personnel for the timeframe category 1) detailed here, and category 2) which follows. The task lists should have timelines for task completions. b) Recommendations on things to-do. - Correctness and robustness of decisions is key, and how to attain and test with beam and silicon at various luminosities should be on the task list. - Demonstrate and test the readout list calculation and transmission to the TS - Demonstrate that dynamic prescaling is working - Demonstrate that the Pulsar is pipelining the assembly and the algorithm processing of events > Buffer based L2 latency, time from L1A to L2Decision as a function of the L2 buffer number, could be used to get an idea of how pipelining is working. For example, L2 buffer 4 is only used when the others are occupied. - Head-to-head comparisons of the Pulsar alone versus the Alpha alone (NOT symbiotic running) under realistic running conditions: beam with silicon and a realistic trigger table at reasonably high luminosity. An Alpha - Pulsar - Alpha run sequence is suggested. > See what the Scalermon report for L2 Latency is (time the TS sees at least one Level 2 buffer full awaiting a L2 decision divided by the number of L1Accepts). > Although not definitive, compare L2 deadtime. - Add a bit to TL2D indicating whether it is from the Pulsar or Alpha. Since the driving system has its block first, it allows the determination of which system is driving from the data. - Add Alpha/Pulsar indications to the ScalarMon window so as to be able to distinguish: i) Which system is providing the L2 rates to the window, and ii) Which system is driving the dynamic prescaling. - Provide a detailed data monitoring plan for: i) PulsarMon > PulsarMon, as part of the shift crew TrigMon, should have a standard slide show folder and an Experts folder. Initially, we suggest that the shift crew's slide show folder be empty, with Expert plots move into it as experience is gained. ii) Offline monitors/validation. > If offline processing is required, it may be useful to make arrangments with the Offline Productions group to quickly process the data taken. - Develop a L3 Filter that flags events where TL2D from the Alpha does not match TL2D from the Pulsar i) Bill Badgett's already existing filter that looks for DAQ related problems can be used. As this already exists, no trigger table changes are required. The filter should be run on every event, and discrepancies written to stream D. Suggestions, ordered from the highest to the lowest priority are: > Compare the decision bits from the two systems and require an exact match with the possible exception of triggers which have a rate limit since they will not necesarily match. Do not exclude the RECES and ISO based triggers because we would like to get the events where these decisions disagree stored for further analysis. > Compare the other data in TL2D other than RECES and ISO and again require exact match. > Compare the ISO data but don't flag as bad if the eta and phi of cluster don't agree with Cluster data. This is how existing ISO trigger code flags these events. > Compare RECES data and save events but apply a logarithmic prescale to limit number (eg save first 100, then 1 in 10 until saved 200, then 1 in 1000 until saved 300....) ii) This tool should be in place prior to the category 2) phase. - Resolve other operational issues i) Trigger database: > Assign responsibility for trigger/algorithm code maintenance in database. > Automated Pulsar firmware version tracking relative to trigger version and provisions to prevent mismatches. ii) Identify all code that needs to be in CVS, and insure that they are in the appropriate code respository. This should also include the trigger-table header files. iii) Fix known run control Operational efficiency problems > Removal of unnecessary diagnostics > Efficient HHRs iv) Trigger GUI: > Incorporate Pulsar versions of all active trigger tables > The Pulsar should be part of the default, production trigger GUI, along with the Alpha v) L2 bit space: implementing and testing Pulsar code to handle the increased L2 trigger bit space vi) Hardware inventory: the Pulsar group has shown that the Pulsar boards are robust, and that there are a sufficient number of them. An inventory for other required pieces of hardware (mezzanine cards, etc.) should be taken and demonstated to be sufficient as well. If not, this should be addressed. 2) Operational integration a) The committee supports and asks the Pulsar group to implement their proposed three week Operational integration plan of > A week of Pulsar symbiotic running, and > Two weeks of the Pulsar driving with the Alpha in auto-reject. A detailed plan should be presented to the CDF Operations group. b) The robustness of the Pulsar must be at the level where the online goodrun bit for the trigger must be "good". c) The L2 Pulsar system should be made operationally ready during this period. The operational plans should lead to full integration into CDF shift crew operations. > Integration of the L2 Pulsar system into the CDF Detector Operations Department, of which an important part is the finalizing of MOUs and who is reponsible for what. > Transformation of PulsarMon and other monitoring tools from commissioning tools to shift crew tools. d) Please consider modifying/replacing the legacy L3 filter, L3_L1L2ErrorFilterModule, as a long term project during/after this period. > This L3 filter, constructed from a stripped down version of TRIGSIM, reconstructs the data in TL2D but does not attempt to reproduce the decision bits. > The problem with this filter is in handling the error rate and in manipulating the filter to handle it. > The filter would run on every L2 accepted event and the output would go to Stream D. Logarithmic prescales could be used to control the rate of error events.