Contact Information (Control Room: 630-840-2080)

Paolo Mastrandrea


Paolo Mastrandrea: paolo.mastrandrea@roma1.infn.it
Office: CDF Portkamps 85-D
Tel: 630-840-3502
Cell: 630-362-7627

Michael Morello

Michael Morello: michael.morello@pi.infn.it
Office: CDF Portkamps 169-L
Tel: 630-840-8697
Cell: 630-362-2336

Massimo Casarsa

Massimo Casarsa: casarsa@fnal.gov
Office: CDF 3rd floor-309
Tel: 630-840-4482
Cell: 630-313-0984

Hyun Su Lee

Hyun Su Lee: hslee@fnal.gov
Office: CDF Trailer 327, UChicago Office
Tel: 630-398-6798
Cell: 630-840-5017

Benedetto De Ruzza

Benedetto De Ruzza: diruzza@fnal.gov, b.diruzza@unicas.it
Office: CDF Portakamps 168-A
Tel: 630-840-8617
Cell: 630-303-6349

Silvia Amerio

Silvia Amerio: silvia.amerio@pd.infn.it
Office: CDF Portakamps 164-L
Tel: 630-840-8401
Cell: 630-457-7074



Login, Reboot, Powercycle

Login

Ways to login to a crate :
setup fer
vxcom crateName (e.g. b0svt01)

OR

setup fer
vxlogin crateName (e.g. b0svt01)

ReBoot

Ways to reboot a crate :

    First login to b0dap30
  1. setup fer
    vxboot b0svt01
  2. setup fer
    vxlogin b0svt01
    reboot
  3. setup fer
    vxcom b0svt01
    Ctrl-x(Ctrl-x means reboot)
  4. ask ACE to shepherd
  5. setup fer
    reset_crate b0svt01
  6. Hardware reset: go to the SVT crate you want to reboot, locate the small "rst" button on the leftmost slot, and press it

  7. The last 2 methods should NOT be used, unless the crate is badly stuck, since it does not disconnect the crate from the RT server and it could leave stuck connection in the RT server that will eventually affect data taking.

Powercycle

How to powercycle :
Before powercycling a crate we must run the following script:

setup fer
vxshutdown b0svt01

This will disconnect the crate from the RTserver. Then go to the SVT crate you want to reboot. The power supply is at the bottom. Switch off, wait until all the lights are off (can be several seconds). Only after that, switch on. Last, press the "rst" button on the leftmost slot of the crate.

MEMO

MEMO



Histograms, SPYMON, mapset/hwset, RunControl , Beam Position, Error messages, SVT on TrigMon

Check histogram

How to check histograms

TrigMon Expert, SVTSPYMON, autoSVTMon.

SPYMON

mapset/hwset

How to update mapset/hwset files

Login on b0gateway. Then:

ksu cdf_svt
setup svtvme -d
cd
Now you are in the cdf_svt home directory. Copy the mapset file you want to install:
cp -i ~carosi/offline_100030_20021001000000.mapset ./
Next commit the mapset file to the database and make if the default for future "config":
mapset put offline_100030_20021001000000.mapset
If you have to modify the mapset and to regenerate the CRC, you should refer to this document http://fcdfhome.fnal.gov/usr/carosi/svtinstallconsts_20021001.html If you want to re-install a previous mapset files, go to this web page file:/cdf/home/carosi/svtmon/what.html, look for the right mapset file (e.g. 4outof5lowpt_20030823174134), then
mapset get 4outof5lowpt_20030823174134
...
mapset put 4outof5lowpt_20030823174134.mapset
The commands for updating the hwset file are similar. Just use "hwset put" and "hwset get"

RC-fakehits

How to use Run Control to test SVT with fakehits


-- how to start
(1) in xterm, type "setup fer"
(2) in xterm, type "rc"
(3) Then you have pop-up "run controler".
-- how to partition
(4) In "run controler"'s top side, you have "partitioning", so you select "partitioning->select patition".
(5) Now, you have "Partition Selector" pop up window.
(6) You may want to choose one of blue partition (free) within 0-7 (hardware partition)
-- how to select the run type
(7) In the run controler top, you have "Parameters", so you select "Parameters->Select Run Configuration" with mouse
(8) Then, you have popup window of "Type Selector" you choose "DaqTest->SVT"
(9) You have bunch of selections for SVT test, but only need to choose "SVT_500HZ"
(10) Select the "SVT_500HZ" and then push the "Select" button.
-- how to set the fakehits
(11) "Parameters->Edit or View run setting"
(12) You have popup wondow for run table setting.
(13) middle of the table you have "parameter" and "value" colums, you need to go to "SvtMode" raw of parameter.
(14) Fill the "hw=fakehits_nobypass map=fakehits verbose" for the Value.
Please note that "hw=fakehits_nobypass" has the beam fit disable in order not to change the last beam position during tests.
(15) and push the keyboard return key.
(16) "File->close" of run table setting
-- how to start the fakehits mode
(17) Push the button, "Partition" in the run control below the "START" green elipse
(18) Then push the "Config" button after "IDLE" elipse (state) was turned green. During this config, we downloaded the constants of memories.
(19) Finally, push the "Activate" button after the "READY" state is turned green.
-- how to stop, rerun the run, and finish the run control
(20) If you push "End" button of run control (left-middle), you can stop the run.
(21) After any "IDLE" state (elipse) turned green, you can restart the run following items (18) and (19).
(22) If you finish the test, you push the "Reset" (right-top) after "IDLE" state, then choose "File->Exit".of run control.
-- If you have "red" state of any elipse.
(23) You have some troubles to use run control, but symptom really depends on the case. Please ask Ace shifts to recover it.

RC-fakehit w/ BEAMfit

How to use Run Control to test SVT with fakehits WITH BEAMFIT enabled

-- before you start make a copy of the current beam position
(1) setup svtdaq
(2) cd $SVTDAQ_DIR/../doc/beam
(3) cp lastbeam.txt "into your favorite temporary area"
-- run it
(4) follow the above procedure
(5) but instead of step (14) do
(6) Fill the "hw=fakehits_nobypass_beamfit map=fakehits verbose" for the Value.
-- at the end
(7) restore the previous lastbeam.txt file

Beam Position

Check Error Messages

How to check error messages

Error messages are in the "ErrorLog" file in the run summary (top line). For the run summary, go to http://www-cdfonline.fnal.gov/daq/runSummary/

SVT on TrigMon

TrigMon for SVT information

TrigMon is running SVTMon and the svtsim. It provides the basic plots to check the SVT performances. The CO is checking those plots for us. So, it is important to make them consistent, to keep references up to date and to add all check plots that we need monitored. Below there are some relevant information about TrigMon.

If you need information about TrigMon in general contact Charles Plager. If you need information about SVTMon running within TrigMon ask Stefano Torre or Roberto Carosi. If you need information about SVTMon comparing SVT to simulation ask Jim Bellinger.

Follow this link to go to the current reference plots. The corresponding files are located here: b0gateway:/cdf/onln/home/cdfdoc/public_html/Consumers/TrigMon/Slides/index.html. If you need to modify them ask Charles Plager to be added to the cdfdoc .k5login file. Please keep them up to date!

The .tcl file for TrigMon sit here: b0gateway:/data1/consumer/runarea/TrigMon/Consumer/TrigMon/runTrigMon.tcl You may need to update it in some cases. For example it would be nice to add to the tcl file threshold to set above what error rate the SVT comparison to simulation should be considered BAD and then should be SVT paged. If you want it changed, make a copy of what you want it to be and send it to Charels.

As per Charles request, do NOT tell information to the CO, instead we should update the documentation. The documentation is the reference plots point above & the "CO's Know Problems" if it is a temporary issue. In the latter case please rember to remove it as soon as the problem is fixed.

The TrigMon log files are located here: b0gateway:/cdf/onln/data/cdfdaq/consumer/results/TrigMon/
The file ErrorLog_LEVEL2_PULSAR_SVT_######.log is useful because it reports mismatches between the data out of SVT and data received by L2. Ask L2 people for more information.



Common Issues

EVB Busy TO

EVB report Busy TimeOut from svt crate

It happend to us (Taichi and Alberto) to get paged because the EVB was going into Busy Timeout. See e-log. The EVB expert looked at the EVB log files and found that the EVB was receiving bad data from crate svt09. We did not know what to do and we asked DAQ to be paged. Ray Culbertson aswered and he suggested to powercycle the crate. This fixed the problem.
So if we get EVB errors due to bad data from SVT crates we should try to powercycle the involved crate. If this does not fix the problem I suggest to page DAQ.

Done TO

Done Timeout (DO) are due to readout going into time out. You should know that readout is not widely used in SVT. We readout the SVTD and SVDD bank from the GB board (crate svt09). We readout timing information for monitoring purposes from the DAQ buffers of TF++ and AMSRW boards (crates svt00-svt06 and svt08).

So if we get DO error from any crate other than svt09 the easiest fix is to just disable the readout and the debug the problem when it does not affect data taking. In order to disable the readout you have to:

If the DO error for svt09, we need to solve the problem. We currently do not have a way to debug this. You can try the usual things such as reboot, powercycle, reseat the tracer and the GB boards. If this does not solve the problem I suggest to page DAQ for help.

SVTSPYMON not running

SVTSPYMON/SpyManager not running ACEs should have instructions on how to bring it back up. Here's a recent link from the e-log: instructions the ace used. Also take a look here for some more specificdetailed instructions.

During Level-2 Torture the ACE notices lots of error messages:(MLE) b0svt06:Messenger:Sep 28, 2005 2:25:29 PM->FER_E_SVT, nwordsMod7=0, EEword=1 in the error log

This is a known, if not understood "feature" of our code. It should go away with real data-taking and can be ignored. Please send email to Alberto if it happens again. With each new version of SVTVME I'm adding debugging info about this problem hoping to understand it sooner or later.

SVT shepherd

SVT crate will not shepherd

Sometimes the SVT crapes shepherd successfully, or are at least in the process of doing so, but with all the ACEs have to remember they forget to watch the front-end console window. When you try to shepherd a crate, a window pops up that shows the progress of the crate. Have the ACE watch this - run control may go red, and the crate is said to have failed, but the crate CPU might just not be done yet. If the shepherd fails, we usually have some form of error message in that terminal window that will detail what board failed and/or what went wrong. If there's no such message, the crate is probably still working - tell the ACE to have patience and keep waiting. This happens often in particular for b0svt08, since the TF++ boards take a long time to do load up memories.

SVT config

SVT crate will not config

What crate was this? If it was b0svt06 or b0svt08, which contain this TF++ boards, try again. If it fails during the Make Address writing (MKADDR) of the TF++ two times in a row, then you need to power-cycle the crate (this memory sometimes has problems being written to, but we don't see normally since this since we check the memory at cold-start and if it reads back as expected we don't need to rewrite it). But don't powercycle anything unless you're sure o the problem.

Again, watch (or have the ACE watch) the front console window to see where things failed - you can open such a window during config by clicking on the crate in RC, clicking open front-end console (towards the bottom of the choices), and then hitting yes to the question you'll be asked about getting a ticket on b0dap10. If it was the MADDR memory, power down the crate, and then power up. It will take a minute or two for the CPU to reboot, and then quite awhile (many, many minutes) while all the memories in the TF++ are reloaded.

If you just loaded new patterns, then all SVT crates will take awhile to config, as there will be many new memories to load. Be patient. In addition, if you power-cycled any crate at all, it will probably take longer than normal to config, since there is more memory to write.

If any board fails to config for reasons other than the TF++ make-address memory noted above, we might need to swap the board. First try configing again (or have the ace try again). If that fails, take a look at the error message in the console window, which should detail which board in the crate you should try swapping with a spare (don't forget to power down the crate before swapping any Pulsar boards!)

Heap Corrupts

Many heap corrupts in a crate

Have the ACE reset and shepherd the crate - that usually solves the problem.

occupancy plot

New hole in SVT occupancy plot

Most likely culprit? Another silicon ladder went dead, and there aren't at least 4 good layers in that barrel/wedge. You can ask to get confirmation of this from the silicon folks - if that's true, there's not much we can do about it.

High deadtime

High deadtime - L2 says it's due to SVT

The first thing to check is if the beam has moved. See above. If it hasn't moved and SVT knows where the beam is located, then this is a more difficult problem to debug.

SVT Beam fit Fails

SVT Beam fit fails:

If the CO has found that SVT cannot find the beam, the first question to ask: Is this a silicon run? If not, of course SVT can't find the beam.

The next thing to ask - did the store just start or did MCR do some tuning? Ask the CO to find the beam position using COT tracks (you can also do this from any PC - go to the CDF e-log, click on the current run, click "Consumers" at the top of the page, and generate BeamMon slides. When it's done, view the beam canvas and find the beam position).

Now compare to the last beam position as found by SVT and also the beam position used to generate patterns and constants, located in the latest mapset file . You want the "active" mapset file. Scroll down to the end of the comments at the top, and there should (hopefully) be a comment describing the pattern center.

OK, so it's a silicon run, and the beam moved. Have some patience - even if the beam has moved quite a bit, the Ghostbuster should be able to find it within the next ~20 or so minutes, at which point CDF can run normally. If the beam is more than ~250 um from the generated position of patterns, or more than ~500-700 um from the position used to generate constants (x and y in quadrature), we need to generate new patterns and constants ASAP and upload them when this store ends. See pager faq page for details. In the meantime, we can probably run OK with the old patterns and constants.

If you want to see something fun, check the d vs phi plot for GB output in the svtspymon histograms for the run before the beam was found. You should see a sinuisoidal pattern.

OK, so it's a silicon run, and the beam did NOT move.