User Guides for Experiments
For directory structure, energy and pole/zero calibration see typical DGS run procedures.
Digital Gammasphere Shift Notes
This section provides a walkthrough of a typical 8-hour shift using DGS. The main workspace looks like this:
General Things to Watch
- Run Control box in DGS Main Controller (top right) must show "Start; Save; Sort" during a run. Otherwise, data is not being recorded.
- TCP/IP rates (center grey boxes) should be changing, and Buffs Avail should be <400 when data is being written to disk.
- VME 01-11 (left white boxes) show values that change in MDIG(n) and SDIG(n) fields. These are Timestamps that should be very close in value when digitizers are all Locked.
- Trigger Rate (bottom right) should be <50 kHz running in "singles" multiplicity (X Threshold = 0). TRIGGER RATE VARIES BY EXPERIMENT DEPENDING ON MULTIPLICITY, BEAM INTENSITY, AND OTHER FACTORS.
The second desktop in the above image (bottom right button) has several white terminal windows, one for each VME (IOC receiver), plus a terminal window with a dgs@dgs1 prompt for start/stop commands. Each VME terminal should update every 15 seconds showing the data in Mb/sec written to disk.
Actions During Shift
It is highly recommended to start a new run approximately every hour. If there is an incident that results in corrupt data for that run, less data would be lost. In /dgsdata directory, stop the current run
./stop_run.sh
Wait 5-10 sec for all Buffs Avail to display 400 in the Big Summary
./start_run.sh ###
where ### is the next run number.
Log in the logbook or elog the stop/start times, current trigger rate, and beam current (or no change in beam current). If you have been advised to check the capacity of the current disk type
df -h
Check Histograms for Quality Control
Digital Gammsphere does not display "live" histograms for users. Users are responsible for checking histograms to ensure sensible data is being collected. Data can be merged, sorted, and displayed in ROOT while it is being collected. See analysis codes for details.
Contact the person ON CALL if data is nonsensical.
Troubleshooting During Shift
This section describes the most common issues that can occur during a shift and how to fix it.
When VME(s) Crash
The first indication of a VME crashing is the TCP/IP rate drops to 0.0 and Buffs Avail continuously fall below 380 without recovering on their own. Stopping the current run and starting a new run before the Buffs Avail reach zero can prevent a crash. If Buffs Avail fall to 0.0, that VME has crashed and needs to be restarted. Check the terminal windows of the IOC receivers and look for an error message (ie something other than Mb/sec written to disk).
- First, stop the current run.
- Second, attempt to reboot the affected receiver by clicking Terminals in DGS Main Controller, then choose the VME (IOC 1 through 11) to be rebooted, which opens a new terminal window.
{insert VME 01 terminal window with reboot countdown here}
- Third, in the terminal window that opens, hit "return" to get a prompt. Then hit Ctrl+X to see a countdown for the IOC to reboot.
Ctrl+X
Wait for the reboot to finish (takes over 1 min) and ignore any error messages (these are non-fatal warnings). A prompt appears when complete, and the window can be closed. If no auto-boot message appears, or there is no prompt at all, then a hard reboot (power cycling the VME crate) is required to reboot the IOC.
- Fourth, click "Scripts" in DGS Main Controller. Run "Lock All Setup" (takes over 1 min) and check the Trigger Summary screen found under "Trigger" in DGS Main Controller. This is also displayed at the bottom of Big Summary and it should look EXACTLY like this:
- Fifth, click "Scripts" and run "Digitizer Setup" script (takes over 1 min). This issues a series of commands setting digitizer channel PVs. At this point, Buffs Avail should read 400 again, and a new run can be started.
Power Cycling the DAQ
If the "soft" reboot was not successful, a hard reboot of the VME crates is necessary, which requires power cycling. If the remote PCU is operational, go to Network Accessible Power Control Units of DGS.
If access to Area 4 is necessary, check the monitor above the cage entrance for room status on ARIS 2.0. The Farday cup must be put "IN" if the area is locked, contact the operators (2-4115 from ANL landline) to request assistance. Click here for the procedure to Sweep Area 4.
Go to the DGS racks (either inside the shack or next to the hemisphere) and cycle the power on the crashed VME crate.
Turn off the crate and wait 30 sec before turning the power back on. Leave the shack and Sweep Area 4.
Back in the Data Room, reboot the Trigger IOC. Click "Terminals" in DGS Main Controller and select Trigger IOC. This is like rebooting a VME IOC, a terminal window opens, hit "return" for a prompt, then use Ctrl+X to initiate the auto-reboot countdown
Ctrl+X
This takes over 1 min. As stated above, ignore any error messages, close the window once the prompt reappears. Next run "Lock All Setup", followed by "Digitizer Setup". Same as stated above.
If there is still a problem, contact the person ON CALL for the shift.
Other Guides for New Users
Go back to Digital Gammasphere Upgrade Project