Michael Morello: email@example.com
Office: CDF Portkamps 169-L
Massimo Casarsa: firstname.lastname@example.org
Office: CDF 3rd floor-309
Hyun Su Lee: email@example.com
Office: CDF Trailer 327, UChicago Office
Benedetto De Ruzza: firstname.lastname@example.org, email@example.com
Office: CDF Portakamps 168-A
Silvia Amerio: firstname.lastname@example.org
Office: CDF Portakamps 164-L
Ways to login to a crate :
vxcom crateName (e.g. b0svt01)
vxlogin crateName (e.g. b0svt01)
Ways to reboot a crate :
How to powercycle :
Before powercycling a crate we must run the following script:
This will disconnect the crate from the RTserver. Then go to the SVT crate you want to reboot. The power supply is at the bottom. Switch off, wait until all the lights are off (can be several seconds). Only after that, switch on. Last, press the "rst" button on the leftmost slot of the crate.
How to update mapset/hwset files
Login on b0gateway. Then:
ksu cdf_svt setup svtvme -d cdNow you are in the cdf_svt home directory. Copy the mapset file you want to install:
cp -i ~carosi/offline_100030_20021001000000.mapset ./Next commit the mapset file to the database and make if the default for future "config":
mapset put offline_100030_20021001000000.mapsetIf you have to modify the mapset and to regenerate the CRC, you should refer to this document http://fcdfhome.fnal.gov/usr/carosi/svtinstallconsts_20021001.html If you want to re-install a previous mapset files, go to this web page file:/cdf/home/carosi/svtmon/what.html, look for the right mapset file (e.g. 4outof5lowpt_20030823174134), then
mapset get 4outof5lowpt_20030823174134 ... mapset put 4outof5lowpt_20030823174134.mapsetThe commands for updating the hwset file are similar. Just use "hwset put" and "hwset get"
How to use Run Control to test SVT with fakehits
-- how to start
(1) in xterm, type "setup fer"
(2) in xterm, type "rc"
(3) Then you have pop-up "run controler".
-- how to partition
(4) In "run controler"'s top side, you have "partitioning", so you select "partitioning->select patition".
(5) Now, you have "Partition Selector" pop up window.
(6) You may want to choose one of blue partition (free) within 0-7 (hardware partition)
-- how to select the run type
(7) In the run controler top, you have "Parameters", so you select "Parameters->Select Run Configuration" with mouse
(8) Then, you have popup window of "Type Selector" you choose "DaqTest->SVT"
(9) You have bunch of selections for SVT test, but only need to choose "SVT_500HZ"
(10) Select the "SVT_500HZ" and then push the "Select" button.
-- how to set the fakehits
(11) "Parameters->Edit or View run setting"
(12) You have popup wondow for run table setting.
(13) middle of the table you have "parameter" and "value" colums, you need to go to "SvtMode" raw of parameter.
(14) Fill the "hw=fakehits_nobypass map=fakehits verbose" for the Value.
Please note that "hw=fakehits_nobypass" has the beam fit disable in order not to change the last beam position during tests.
(15) and push the keyboard return key.
(16) "File->close" of run table setting
-- how to start the fakehits mode
(17) Push the button, "Partition" in the run control below the "START" green elipse
(18) Then push the "Config" button after "IDLE" elipse (state) was turned green. During this config, we downloaded the constants of memories.
(19) Finally, push the "Activate" button after the "READY" state is turned green.
-- how to stop, rerun the run, and finish the run control
(20) If you push "End" button of run control (left-middle), you can stop the run.
(21) After any "IDLE" state (elipse) turned green, you can restart the run following items (18) and (19).
(22) If you finish the test, you push the "Reset" (right-top) after "IDLE" state, then choose "File->Exit".of run control.
-- If you have "red" state of any elipse.
(23) You have some troubles to use run control, but symptom really depends on the case. Please ask Ace shifts to recover it.
How to use Run Control to test SVT with fakehits WITH BEAMFIT enabled
-- before you start make a copy of the current beam position
(1) setup svtdaq
(2) cd $SVTDAQ_DIR/../doc/beam
(3) cp lastbeam.txt "into your favorite temporary area"
-- run it
(4) follow the above procedure
(5) but instead of step (14) do
(6) Fill the "hw=fakehits_nobypass_beamfit map=fakehits verbose" for the Value.
-- at the end
(7) restore the previous lastbeam.txt file
How to check error messages
Error messages are in the "ErrorLog" file in the run summary (top line). For the run summary, go to http://www-cdfonline.fnal.gov/daq/runSummary/
TrigMon for SVT information
TrigMon is running SVTMon and the svtsim.
It provides the basic plots to check the SVT performances.
The CO is checking those plots for us.
So, it is important to make them consistent, to keep references up to date and to add all check plots that we need monitored.
Below there are some relevant information about TrigMon.
If you need information about TrigMon in general contact Charles Plager. If you need information about SVTMon running within TrigMon ask Stefano Torre or Roberto Carosi. If you need information about SVTMon comparing SVT to simulation ask Jim Bellinger.
Follow this link to go to the current reference plots. The corresponding files are located here: b0gateway:/cdf/onln/home/cdfdoc/public_html/Consumers/TrigMon/Slides/index.html. If you need to modify them ask Charles Plager to be added to the cdfdoc .k5login file. Please keep them up to date!
The .tcl file for TrigMon sit here: b0gateway:/data1/consumer/runarea/TrigMon/Consumer/TrigMon/runTrigMon.tcl You may need to update it in some cases. For example it would be nice to add to the tcl file threshold to set above what error rate the SVT comparison to simulation should be considered BAD and then should be SVT paged. If you want it changed, make a copy of what you want it to be and send it to Charels.
As per Charles request, do NOT tell information to the CO, instead we should update the documentation. The documentation is the reference plots point above & the "CO's Know Problems" if it is a temporary issue. In the latter case please rember to remove it as soon as the problem is fixed.
The TrigMon log files are located here: b0gateway:/cdf/onln/data/cdfdaq/consumer/results/TrigMon/
The file ErrorLog_LEVEL2_PULSAR_SVT_######.log is useful because it reports mismatches between the data out of SVT and data received by L2. Ask L2 people for more information.
Done Timeout (DO) are due to readout going into time out.
You should know that readout is not widely used in SVT.
We readout the SVTD and SVDD bank from the GB board (crate svt09).
We readout timing information for monitoring purposes from the DAQ buffers of TF++ and AMSRW boards (crates svt00-svt06 and svt08).
So if we get DO error from any crate other than svt09 the easiest fix is to just disable the readout and the debug the problem when it does not affect data taking. In order to disable the readout you have to:
SVTSPYMON/SpyManager not running
ACEs should have instructions on how to bring it back up. Here's a recent link from the e-log: instructions the ace used. Also take a look here for some more specificdetailed instructions.
During Level-2 Torture the ACE notices lots of error messages:(MLE) b0svt06:Messenger:Sep 28, 2005 2:25:29 PM->FER_E_SVT, nwordsMod7=0, EEword=1 in the error log
This is a known, if not understood "feature" of our code. It should go away with real data-taking and can be ignored. Please send email to Alberto if it happens again. With each new version of SVTVME I'm adding debugging info about this problem hoping to understand it sooner or later.
SVT crate will not shepherd
Sometimes the SVT crapes shepherd successfully, or are at least in the process of doing so, but with all the ACEs have to remember they forget to watch the front-end console window. When you try to shepherd a crate, a window pops up that shows the progress of the crate. Have the ACE watch this - run control may go red, and the crate is said to have failed, but the crate CPU might just not be done yet. If the shepherd fails, we usually have some form of error message in that terminal window that will detail what board failed and/or what went wrong. If there's no such message, the crate is probably still working - tell the ACE to have patience and keep waiting. This happens often in particular for b0svt08, since the TF++ boards take a long time to do load up memories.
SVT crate will not config
What crate was this? If it was b0svt06 or b0svt08, which contain this TF++ boards, try again. If it fails during the Make Address writing (MKADDR) of the TF++ two times in a row, then you need to power-cycle the crate (this memory sometimes has problems being written to, but we don't see normally since this since we check the memory at cold-start and if it reads back as expected we don't need to rewrite it). But don't powercycle anything unless you're sure o the problem.
Again, watch (or have the ACE watch) the front console window to see where things failed - you can open such a window during config by clicking on the crate in RC, clicking open front-end console (towards the bottom of the choices), and then hitting yes to the question you'll be asked about getting a ticket on b0dap10. If it was the MADDR memory, power down the crate, and then power up. It will take a minute or two for the CPU to reboot, and then quite awhile (many, many minutes) while all the memories in the TF++ are reloaded.
If you just loaded new patterns, then all SVT crates will take awhile to config, as there will be many new memories to load. Be patient. In addition, if you power-cycled any crate at all, it will probably take longer than normal to config, since there is more memory to write.
If any board fails to config for reasons other than the TF++ make-address memory noted above, we might need to swap the board. First try configing again (or have the ace try again). If that fails, take a look at the error message in the console window, which should detail which board in the crate you should try swapping with a spare (don't forget to power down the crate before swapping any Pulsar boards!)
Many heap corrupts in a crate
Have the ACE reset and shepherd the crate - that usually solves the problem.
New hole in SVT occupancy plot
Most likely culprit? Another silicon ladder went dead, and there aren't at least 4 good layers in that barrel/wedge. You can ask to get confirmation of this from the silicon folks - if that's true, there's not much we can do about it.
High deadtime - L2 says it's due to SVT
The first thing to check is if the beam has moved. See above. If it hasn't moved and SVT knows where the beam is located, then this is a more difficult problem to debug.
SVT Beam fit fails:
If the CO has found that SVT cannot find the beam, the first question to ask: Is this a silicon run? If not, of course SVT can't find the beam.
The next thing to ask - did the store just start or did MCR do some tuning? Ask the CO to find the beam position using COT tracks (you can also do this from any PC - go to the CDF e-log, click on the current run, click "Consumers" at the top of the page, and generate BeamMon slides. When it's done, view the beam canvas and find the beam position).
Now compare to the last beam position as found by SVT and also the beam position used to generate patterns and constants, located in the latest mapset file . You want the "active" mapset file. Scroll down to the end of the comments at the top, and there should (hopefully) be a comment describing the pattern center.
OK, so it's a silicon run, and the beam moved. Have some patience - even if the beam has moved quite a bit, the Ghostbuster should be able to find it within the next ~20 or so minutes, at which point CDF can run normally. If the beam is more than ~250 um from the generated position of patterns, or more than ~500-700 um from the position used to generate constants (x and y in quadrature), we need to generate new patterns and constants ASAP and upload them when this store ends. See pager faq page for details. In the meantime, we can probably run OK with the old patterns and constants.
If you want to see something fun, check the d vs phi plot for GB output in the svtspymon histograms for the run before the beam was found. You should see a sinuisoidal pattern.
OK, so it's a silicon run, and the beam did NOT move.