Guide to Validating and Debugging the Level 1 Trigger

General

  • Check the TFRD Monitor - Bunch Counter plot in TrigMon to ensure that triggers are spread evenly across all filled bunches. This can also be checked on the "TS" tab of the scaler GUI which shows the number of Level 1 Accepts per bunch. These frequencies should correlate with the luminosity per bunch. If they don't, there could be something wrong with the trigger, the Trigger Supervisor, or the clock.

  • The bunch counter values should agree for all detectors. Check this in the BCSS Monitor plots in TrigMon. These histograms should always be empty!

    Global Level 1

  • Watch this space - FREDSim coming soon.

    L1 Calorimeter

    Possible scenarios:

      (1) There are no triggers

      Easy to spot, harder to fix.

      (2) There are triggers on every crossing

      Easy to spot, easier to fix.

      (3) The trigger is performing perfectly. How to tell?

      Mostly for experts but have a look!

    (1) There are no triggers

    This almost never happens now but here is a list of things that have happened in the past.

    • The calorimeter high voltage is not turned on. Check the analog meters on the first floor and/or the HVmon display.

    • The E_T weighting tables in the ADMEM cards have been wiped out.
      If you suspect this is the case, run control and calorimeter experts should be called.

    • All trigger channels have been accidently masked at the DIRAC or PFRED stage due to a database snafu. This can be checked using the DIRAC and PFRED GUIs in the fdaq teststand or by examining the run control configuration messages.
      If you think this is the problem you should contact the run control Hardware Database expert.

    (2) There are triggers on every crossing

    This could be due to problems with the calorimeter or the trigger hardware.

    • Check the calorimeter lego plot in the Event Display. You may see one or more towers on all the time. This could be due to a bad calibration, a light leak, etc. Report it to the calorimeter group.

    • Now check the trigger hardware. Possible problems are:

      • A loose or disconnected trigger cable that causes spurious triggers.
        The TRGSim++: Dirac Tot Triggers Occupancy plot shown below indicates such a loose cable.

        One can mask the offending channel with traceDirac (requires Oracle password). Only mask towers triggering too frequently; hot PMTs that don't cause triggers don't need to be masked.

      • An error in downloading the ADMEM boards. For example, if the trigger tower pedestals are set to zero this can cause an abnormal trigger rate, depending on the trigger table. Such a problem would also be seen in the TRGSim++: Dirac Tot Triggers Occupancy plot but for many towers.

        These types of problems can be further identified by checking the "Trigger cables/DIRAC/DCAS" section of the L1TriggerSummary_XXXXXX.log file (where XXXXXX is the run number) in the directory /data1/consumer/results/TrigMon. An expert can also compare the trigger tower occupancies to Admem occupancies inside the L1TriggerSummary_XXXXXX.log file to decide if the problem is in the calorimeter or the trigger.

        If an ADMEM channel has entries but the corresponding trigger tower does not and it is not masked, this would be noteworthy and should be reported it in the log book. It could indicate a missing DCAS card or a bad cable.

    • Spikes! These will show up in the ADC lego plot of the Event Display. If the spikes are confined to a given channel(s), proceed to mask as above.

    (3) Is the trigger performing well?

    • Look at ADMEM and trigger tower occupancy: you should see entries corresponding to all crates booked. The occupancy pattern depend on the type of data and trigger table used, but there are typical ones....

    • TrigMon has information on consistency at all steps of the trigger chain:

      • ADMEM-Trigger Towers: the ADMEM cards perform pedestal subtraction, channel summing, ADC counts to GeV conversion and ET weighting to calculate trigger tower energies sent to the trigger. Check expected values vs trigger readout in the TC2D bank for the three calorimeter types.

        ADMEM Canvas 1 shows trigger tower energy vs exepcted ones for EM(left) and HAd(right)- top two plots, and the ratio of the two as function of trigger eta, again for EM and HAD, btoom plots. Events liying on lines parallel to the diagonal in to the top plots indicate stuck or dead bits either in the input to Dirac or in Dcas. The bottom plots check the Et weighting and scale factors across the calorimeter types.

        ADMEM Canvas 2 shows the (eta,phi) map of towers disagreeing with expectation and difference vs tower energy in counts, for EM(left) and Had(right).

        If they disagree:

        • is problem confined to a detector/crate/card in the ADMEM or trigger system? Look at plots above of failures, ratios of trigger_tower/Admem as function of eta. Only disagreement of >=4 bits will affect electron/photon triggers, >=8 bits will affect jet/sumet/met triggers.

      • DIRAC: there are 96 Diracs, each four of them covering a full wedge. Each Dirac card sums Et energies for six eta channels in given wedge, EM and HAD seperately, sets bits corresponding to single tower or two towers above thresholds defined in the trigger table, and to photon and electron triggers. The distribution of bits on for each Dirac is shown in L1TriggerSummary_XXXXXX.log. Is the Dirac result consistent with its input?

      • DIRAC-CRATESUM: 12 Cratesum boards make summaries of trigger information from 8 Diracs each, both for energy and trigger bits. Look in L1TriggerSummary_XXXXXX.log for summary of checks.

        ************ Check Et between Dirac and Cratesum:
        CS1: 0 0 0 0 0 0 0 0 0 0 0 0
        CS2: 0 0 0 0 0 0 0 0 0 0 0 0
        CS1 and CS2 refer to the first and second set of Diracs seen by a Cratesum, the twelve set of numbers refer to each Cratesum, indicating the number of events where there is a disagreement.

        Cratesum Canvas 1 shows agreement between the Cratesum output with the one expected from the input received by Dirac, bottom plot, and using as input the values as read from Dcas. A disagreement in the trigger tower/Admem plots will show up in th the top plot, as in the example here. No entry in the corresponding location in the bottom plot indicates that the Cratesum works correctly with the input it receives. Each cratesum serves two wedges, the comparison is done by wedge-> two entries/cratesum.

        Cratesum Canvas 2 maps which of the eight Dirac corresponding to a Cratesum doesn't agree with:

      • PFRED: look at the trigger bits distribution: should agree with what expected from the trigger table.

        Two Pfred boards are involved in the Calorimeter trigger:

        • SUMET calculates the total Sum_Et, its x and y components, and MET, setting 2 trigger bits if Sum_Et is above threshold, and 2 bits for MET. The bits are sent to FRED, Sum_Et,Et_x, Et_y are sent to Level 2. The Pfred results should agree with the Cratesum inputs.

        • CALTRG makes the trigger bits summary for FRED. Again it should agree with the Cratesum inputs. For both Pfreds check the Pfred/Cratesum section in L1TriggerSummary_XXXXXX.log

          ********** Pfred/Cratesum :

          Sumet mismatch between Pfred and Cratesum 0

          Cratesum in bad events: 0 0 0 0 0 0 0 0 0 0 0 0

          Caltrg mismatch between Pfred and Cratesum 0

          Cratesum in bad events: 0 0 0 0 0 0 0 0 0 0 0 0

          A non-zero entry in the list above will indicate wich cratesum had data in the events in error.

          ********************************************************************************

      • GLOBAL: Check Sum_Et and MET plots, should show current thresholds,MET^2 vs Et_x^2+Et_y^2, Sum_Et in Pfred vs sum of ADMEM energies.

      • FRED: Check agreement with the trigger bits sent by Pfred, prescale values.

      TRIGGER Efficiencies In progress...
      carla@fnal.gov
      feild@fnal.gov