Operations

To Start the DAQ under VNC

instructions 3/6/2021 (from Claus)

on WS3, from gretina account, start the VNC server and make a local VNC viewer

 vncserver -list                          should give you a list of all running sessions
 vncserver -kill :3                       will kill the running session as it may be broken
 rm /tmp/.X11-unix/X3                     delete a hidden file, if the kill command doesn't work
 vncserver :3                             start a new session
 vncviewer :3                             opens the viewer locally, you can also follow instructions to connect remotely

now you should do these things in the VNC (which is local on WS3):

 ssh -X a2
 xset fp+ /usr/share/X11/fonts/75dpi      fixes a font problem, otherwise EPICS control will not start
 ./startGRETINA

that should bring some of the GT DAQ up. Now setup the DAQ

 kill DAQ
 initialize DAQ
 GRETINA+FMA

now the runcontrol should come up and you can start a run. Note: now you have to start and stop runs from a terminal as:

 gDAQon
 gDAQoff

It increments the run number itself.

next

 firefox
 connect to the data room blue jeans
 share your screen

Now people on the dataroom BJ can see what you see without having to make their own VNC connection

If you are also running the DFMA DAQ, you should follow the instructions below to get that DAQ going as well.

In a terminal window (5)

 ssh -X gretina@dmfa_darek
 ./dfmaCSS
 check all IOCs are up or cycle power
 get to trigger IOC and reboot
 on DFMACommander.opi
   scripts
   setup_system_sync

now DFMA should be up; but not synced to GT yet. To do that

 got to GT DAQ
 stop run
 Syncro FIX
 start run

Now, in a terminalwindow (4):

 ssh -X gretina@dmfa_darek
 cd /global/data1a/gretina/exp1861_Lotay_data1a/dfmadata

here you can

 start_run.sh xxx
 stop_run.sh

being in sync with the GT DAQ regarding the run numbers

To Start the DAQ

[historical entry]

issue this command to get the fonts necessary for EDM

• xset fp+ /usr/share/X11/fonts/75dpi

• There are two ways to start the Gretina data acquisition system, depending on the computer in use;

1. On the iMac: Double click on the Gretina icon (start_DAQ) located on the top left corner of the iMac's desktop and the window (shown to the right --->) will pop up.
2. On the WS2 and WS3 Linux machines: First logon to a1.gam or a2.gam using the commands below

ssh -X [email protected]

or

ssh -X [email protected]

Once in the home directory ~/, type the command below to bring up the Gretina DAQ GUI (shown to the right --->).

 ./gretinabuttons.tcl 

To make sure the DAQ is not running anywhere else, first PRESS

 KILL DAQ

Wait until you see in the lower shell: stop. Everything should close now. Then PRESS

INITIALISE DAQ

This will take a few moments, up to 1-2 minutes. Observe the lower shell and check that it is finally stating: READY FOR CALIBRATION

In this mode, GRETINA is ready for data taking in calibration mode, meaning: validation mode, 0.2us traces. Please start a run, if you just want to make a calibration.

If you want to couple GRETINA to CHICO [or run it standalone] in DECOMP mode, press either

GRETINA STAND ALONE 

or

GRETINA+CHICO 

Again, things will happen in the shell and it is loading bases and setting up the triggers.

NOTE: the only difference between GRETINA STAND ALONE and GRETINA+CHICO is the trigger mask. In order to enable an external trigger and to disable the internal triggers, the ROUTERS will be changed correspondingly.

Now: GRETINA is set up to the correct mode and the CLUSTER STATE [in CLUSTER → NODES] should be ok. If not, please wait for a certain amount of time [could be more than 5 minutes]. If this does not help: try the full procedure again.

 You may want to go to the Cluster page and select the proper data 
 format, either 'mode2/mode3' or just 'mode2'. The default seems 
 to be 'mode2/mode3'. Note: you should do that before you take any data.

To Start the DAQ (old way)

[historical entry]

• We run the DAQ from "a1.gam" (for now)

ssh -X [email protected] 
or 
ssh -X [email protected]

• you may have to issue this command to get access to the proper fonts for the EPICS displays

 xset fp+ /usr/share/X11/fonts/75dpi

• Go to the gretina home directory

 cd
 source goatlas

• The MENU below should show up:

---------- MENU ----------
  VME    	     1
  Display	     2
  Run Control	     3
  Run ControlBGS    3b
  Soft IOC	     4
  Soft IOC2	     4b
  CAEN PS	     5
  Cluster           6
  Cluster Kill      8
  Mode 2	     9
  Mode 3	     10
  Mode 4	     11
  RESET daq InBeam  12
  RESET daq Gamma   12b
  StripTool	     13
  Calibrate	     14
  
  EXIT  	     E
  KILL ALL	     K

---

• Info: Exiting the MENU does not exit the DAQ. One can return to the MENU anytime by typing $PWD/godaq

• Start things in the following order:

K, 4, 4b, 2, 3, 6

K to make sure something is not running somewhere. After this sequence, verify you see "CA Connected" (CA: Channel Access) in the RunControl panel

• Press Button "cluster" in the GlobalControl panel to chose the correct data mode (mode 2: decomposed data, mode 3: raw data (traces)). If GT is not set up already for mode2 data taking (producing decomposed data) you should invoke option 9.

Note: In the Gretina RunControl panel, 'new dir' makes the run starts at 0002, switching back to an existing directory makes the run start at the previous Run number + 2 

• Press button "global control"

The upcoming panel is being used for setting global 
parameters for ALL chrystals (baseline, polarity, etc.)

    Raw delay: For internal mode data: 3.7 usec (for analyzing the traces)
               typically you will see 2.7 usec here

    Raw len:  length of the trace in usec 
          don't go below 0.2 usec (corresponding to 20 
          short words, while the header is already 16 words long, 
          so there are 4 words left for the trace itself)
          in decomposition mode: Use 2.0 usec 
          (corresponding to 16 words for the header and 184 
          words for the trace)

    Threshold setting: typically cc: 1000, segment 300 (do not go below 300
      in the segments and 150 in CC)
    
    in "user trigger" you can set the crystal readout rate up to 20,000. 
    Lower this number if the decomposition cluster cannot keep up

• Press button "TimeStamps" in the Gretina DAQ top level to check if the timestamps are still synchronized imp sync: starts a train of imperative sync signals, by pressing the button again, the train stops single pulse imp sync: One single imperative sync pulse

• press button "pseudoScalers" in the Gretina DAQ top level to verify that data is coming in and being processed

• rebooting digitizer IOCs

 ssh gsoper:[email protected]
 find the number in the "timestamp window". simply count to 
 get the xx value to use. For crate 3, use 03. Do not ssh from 
 the terminal you set up GT from as you will need to type "~."
 to get out and that will kill your window as well. Always ssh 
 from a seperate terminal window

 if you telnet into the IOC and you see tasks suspended (type i
 to see the processes and thir status) you need to reboot. Simply 
 type "reboot". Using "tr" to start them usually does not help

 after a reboot: push the "reset after an IOC reboot" butten 
 and the "TTCS settings button" and the other buttons there.

for access to the trigger IOC,

 ssh gsoper:[email protected]

Once all is set up, you can also start and stop runs on the command line with these commands (scripts in ~/scripts)

 gDAQon  (does not work?)
 gDAQoff (works)

The alarm page is accessed through the GT main DAQ screen upper right corner. This is where you can mute the alarm

to stop the DAQ

push the stop button in run control window

If you want to stop everything, select "K" in the $PWD/godaq script.

to shut down the DAQ

• Type 8 in the MENU to kill the Cluster

• Type K (Kill All) in the MENU to kill the rest

online sorting with GEBSort, to a map file

NOTE: you can only receive on-line data if the decomposition cluster is running. The Global Event Builder, where GEBSort gets it data from, is only active if the decomposition cluster is up and running

Here is an example of running GEBSort on ws3.gam taking data from the GEB to a map file we can look at while we take data

o ssh -X [email protected]

o Download [THIS IS OUT OF DATE]

 cd workdir (e.g., /home/gtuser/gebsort)
   https://svn.anl.gov/repos/gs_analysis/GEBSort .
 or
   wget http://www.phy.anl.gov/gretina/GEBSort/AAAtar.tgz
   tar -zxvf AAAtar.tgz

on a1.gam you would need to do this as well

 export ROOTSYS=/home/users/tl/root/root-v5-34-00
 export PATH=$ROOTSYS/bin:$PATH
 export LD_LIBRARY_PATH=/home/users/tl/root/root-v5-34-00/lib/root

on ws3 or ws2 you should be fine

o now compile everything

 rm curEPICS; ln -s /global/devel/base/R3.14.11 curEPICS
 make clean
 make 
 make GEBSort
 ./mkMap > map.dat
 rm GTDATA; ln -s ./ GTDATA

o to run and take data, type

#               +-- Global Event Builder (GEB) host IP (or simulator)
#               |           + -- Number of events asked for on each read
#               |           |  +-- desired data type (0 is all)
#               |           |  |  +-- timeout (sec)
#               |           |  |  |   
 ./GEBSort \
   -input geb 10.0.1.100  100  0 100.0 \
   -mapfile  c1.map 200000000 0x9ef6e000 \
   -chat GEBSort.chat 

you may also just say

 go geb_map

o to display the data in the map file

 rootn.exe -l 
 compile GSUtils (first  line)
 sload c1.map file (third  line)
 update          (fourth line and other places)
 display

o note: It can take a while, up to a minute or so, before the GEB has data to offer to ./GEBSort so if ./GEBSort doesn't get data, just try again after you have started a run.

off-line sorting with GEBSort, to a root file

Here is an example of running GEBSort with output to a root file in an off-line situation

o Download the software

 cd workdir (e.g., /home/gtuser/gebsort)
 git clone https://gitlab.phy.anl.gov/tlauritsen/GEBSort.git .

o now compile everything

 make clean
 make 
 ./mkMap > map.dat
 rm GTDATA; ln -s ./ GTDATA

Ignore errors about not being able to make 'GEBSort' if you don't have VxWorks and EPICS installed! It will make 'GEBSort_nogeb', which is the version of GEBSort we will use for off-line data sorting where VxWorks and EPICS are not needed (the on-line situation, where you DO need GEBSort, is described above).

o sort your (merged) file as:

  ./GEBSort_nogeb \
   -input disk merged_data.gtd \
   -rootfile test.root RECREATE \
   -chat GEBSort.chat 

note1: for DGS data, GEBSort_nogeb works equally well on the idividual files from gtReceiver4 as the merge of those files created with GEBMerge

o to display the data in the root file

 rootn.exe -l  (or 'root -l' since it is just a root file)
 compile GSUtils (first  line)
 dload test.root file (third  line)
 display

In general, the 'go' file of the GEBSort package contains a lot of examples. You can make your own script file to process your data using these examples.

misc problem fixes and procedures

• To fix a non-responsive timestamp, i.e. just counts and never resets, use command "caput Cry[#]_CS_Fix (e.g., caput Cry4_CS_Fix 1) 1" where [#] is the bank # of the offending timestamp(s). This will reload the FPGA code and reboot the IOC. Any time an IOC is rebooted, you should press the "RESET after an IOC Reboot" button on Global Controls edm screen. Then choose the settings type you want, usually TTCS or Validate settings, on the same screen. Note: this only works if the IOC is still respnding to CA. If it doesn't, try access through the terminal server and reboot, or cycle the power on the crate. Typically you have to manually reboot when tExcTask and tNetTask are suspended.

• to select the decomposition nodes that should be runnin (and, thus, avoid any nodes that are broken), edit the files: "setup7modulesRaw.sh", "setupEndToEnd28.sh" in the directory: "/global/devel/gretTop/11-1/gretClust/bin/linux-x86". Then do 8 to kill , then run the scripts you just changed, and 6 to bring up decomposition cluster up again. On dogs, type 'wwtop', to see details of what nodes are running.

• to bring up the GT repair buttons: "~/gretinabuttons.tcl". you may have to do that from a1.gam for now as we have a problem doing some things from the linux WS machine;

• If some crystals are not getteing decomposed or running properly: stop, do imp sync, and start again. If that is not enough, also click on the 4 buttons in the south east corner of the "global Control" page. If it still does not decompose, stop the decomposition cluster and start it again in the main DAQ script

• set the buffer size in the GEB, use

  caput GEB_CS_Delay 5

in this case, we set it to 5 seconds. 5 to 10 are good values.

• another mysterious thing to try

 LEDAT 19 700;LEDAT 23 800;SetPZMultCC.sh;SETLEDWIN 30

Calibration Procedure ANL -- version

get the data

To obtain the data needed to do the segment and CC calibration, we change the GT configuration to

Raw Len = 0.20 us
Trig Mode = Validate
mode 3 data (in godaq script)
the cluster nodes should be running 1 sendRaw process per crystal
in cluster, select 'mode 2 and 3'

you may have to adjust 'Crystal readout Rate Limit' to adjust the rate. In this mode you can take data at a rate of 60 times faster than when you run your data through the decomposition cluster to obtain mode2 data. You can split the files using this handy utility

 gMultiRun <number of runs> <minutes per run> <minutes between runs>

check that the

 GlobalRaw.dat

file increments. There is also a Global.dat, but it should be empty.

run the calibration software

Shaofei is working on this software. Eventually we will add it to GEBSort, where the date will be processed in the 'bin_gtcal' module.

Calibration Procedure -- NSCL version

NOTE: these instructions were lifted from NSCL, we will have to modify them to the needs at ANL

In broad strokes, the procedure has been to fit source spectra for the central contacts (CC), and then calibrate the segments based on the observed energy in a segment vs. the calibrated CC energy for multiplicity 1 events. There are some simple, and fairly quick unpacking codes which make the necessary ROOT spectra, and then some ROOT macros that actually perform the fitting etc. The procedure is outlined below, and the codes are all located in /home/users/gretina/Calibration/.

CC Calibration

1. Unpack the calibration data (this all assumes it's mode3 data) using the code ./UnpackRaw, where the commands are

     ./UnpackRaw  <Run#, i.e. 0061>

2. Run the macro to fit the peaks:

     root -l
     root [0] .L FitSpectra.C
     root [1] FitCCSpectra(.root Filename, i.e. "MSUMay2012Run0060.root",
                           value for xmin, typically choose 100,
                           Boolean for manual checking, to check, set to 1, otherwise 0,
                           source type, i.e. "Eu152");

-- Here the inputs are the .root filename (i.e. "MSUMay2012Run0061.root") which was generated in the unpacking step, a minimum spectrum value to look at (not so important, just pick something low, like 100 or something), a boolean to say if you want to personally approve the fits (CHECK = 1), or just let it go (CHECK = 0), and the source type (options are "Co60", "Bi207", "Eu152", "Co56" and "Ra226" right now).

-- If you run this script with CHECK == 1, you run in a quasi-interactive mode. That is, for each CC, you'll get a chance to look at all of the fits, and to move to the next CC, you must accept the fits by hitting 'y', and then enter. If you see one you don't like, you can enter the peak number, and the fit will be redone, with tweaked parameters. Then you can either accept the peak, or reject it, in which case it's ignored in fitting the calibration. Peaks are numbered in order of increasing energy, and labeled in the ROOT canvas, so you know which number to type for a given fit. The only thing to note in this is that if peaks are < 20 keV apart, they are fit together. In this case, if peaks 7 + 8 are fit together, there will only be a spectrum shown in the position corresponding to peak 7. If you're unhappy with the fit though, and refit 7, it will only fit peak 7. If you want peak 8 retried as well, you need to specify this additionally.

-- Regardless of the value of CHECK, if the code can't find a peak, it'll ask you if you see one. If you do, you can enter the approximate channel number, and a fit will be attempted. If you don't, entering -1 will just leave out the calibration point.

-- Output from the macro:

  paramCC<ROOTFilename>.dat (i.e. paramCCMSUMay2012Run0061.dat) -- contains all fit parameters
  meanCC<ROOTFilename>.dat (i.e. meanCCMSUMay2012Run0061.dat) -- contains just the centroid of the fits

3. After you have fit all the source spectra you're interested in, you can fit the calibration with the ROOT macro:

     root -l
     root [0] .L FitCalibration.C
     root [1] FitCalibration(number of calibration files, i.e. 3, 
                             First .root file, i.e. "MSUMay2012Run0060.root", 
                             Second .root file, i.e. "MSUMay2012Run0071.root", 
                             Third .root file, i.e. "MSUMay2012Run0070.root",
                             Fourth .root file, i.e. "" if only using three files,
                             Flag for CCONLY = 1 if only fitting CC,
                             Flag for making detector maps, DETMAPS = 0 for NO);

-- This literally fits the calibration points with a straight line, and calculates the residuals, etc. of the fit. Input to the macro is the number of files you want to use in the calibration (number), the .root filenames (i.e. "MSUMay2012Run0061.root"), a boolean to say if you only want to fit the CC (answer should be yes, so CCONLY = 1), and a boolean to see if you want to make the detector map files for decomposition (at this stage with only CC calibrations, no). Note, if you use less than 4 calibration files, for the unused filenames, just put "".

-- Output from the macro:

  fitFile<ROOTFilename#1>.root 
  (i.e. fitFileMSUMay2012Run0061.root) 
  -- contains the calibration fit, and residual plots

  finalCalOutput<ROOTFilename#1>.dat 
  (i.e. finalCalOutputMSUMay2012Run0061.dat) 
  -- contains three columns - ID, offset and slope
  This is the file I rename (EhiGainCor-CC-MMDDYYYY.dat) and use for CC calibration in unpacking codes.

  finalDiffOutput<ROOTFilename#1>.dat (i.e. finalDiffOutputMSUMay2012Run0061.dat) 
  -- contains residual information in text format

Segment Calibration

1. Edit the unpack-m1.C source code (in the subdirectory src/), and change the line that reads in the calibration to read the new CC calibration that you've created.

     RdGeCalFile("<Name of CC calibration - i.e. finalFitFile<>.dat>", ehiGeOffset, ehiGeGain);

2. Remake the code, just type make. This replaces the executable in the directory above.

3. Run the code UnpackM1 on the data you're using for the segment calibration. Typically, pick something with statistics enough to see something in the back segments.

     ./UnpackMult1 <Directory, i.e. MSUMay2012> <Run Number, i.e. 0061>

-- This code takes a while, because it scans the file 28 times, to calibrate each crystal one at a time. This is because it's using 2D ROOT matrices, and I can't make the memory usage work properly to do multiple crystals at the same time. It always has a memory allocation problem. If someone knows how to fix this, by all means, please do. Let Heather know how you fix it too =)

-- The code is essentially sorting segment multiplicity 1 events, and using the calibrated CC to calibrate the segment energies. Right now, it's looking at the 5MeV CC to do this. You can change this in the code, if desired. At the end of the code, it then fits the plot of CC vs. segE, and extracts a slope and offset. This is actually then written out as a completed calibration file (see output information).

-- Output files from code:

   <ROOTFilename>Crystal##.root 
   where ## goes from 1 to 28 -- these are .root files
  containing the 2D plots for each crystal, as well as projections

   <ROOTFilename>.slope 
   -- contains channel ID (electronics ID), offset and slopes 
   -- the finished calibration 
   This is the file that is usually renamed to 
   EhiGainCor-MMDDYYY.dat and used in codes.

Detector Map and Trace Gain Generation for Decomp

1. Edit the ROOT macro file FitCalibration.C, specifically the function MakeDetectorMaps(), to read in the appropriate calibration file. Also edit the destination directory for the generated detector map files, which appears in a few places...

    calibrationFile = fopen("<CalibrationFilename.slope>", "r");
    ...
    system("mkdir <DestinationDirectoryName>");
    ...
    detmap_file = fopen(Form(<DestinationDirectoryName>/detmap_Q%dpos%d_CC%d.txt", quad, position, ccnum), "w");
    ...
    tracegainfile = fopen(Form(<DestinationDirectoryName>/tr_gain_Q%dpos%d_CC%d.txt", quad, position, ccnum), "w");

2. Run the MakeDetectorMaps() macro.

    root -l
    root[0] .L FitCalibration.C
    root[1] MakeDetectorMaps()

3. Detector map and trace gain files should be automatically generated in the directory specified within the macro.

Statistics required for calibration

 * 56Co,  mode 3, validate, trace 0.2us --- 8 GB = ~15 minutes
 * 152Eu, mode 3, validate, trace 0.2us --- 6 GB = ~15 minutes
 * 226Ra, mode 3, validate, trace 0.2us --- 8 GB = ~15 minutes

Statistics required for checking segments resolution

 * 60Co (10 uC),  mode 3, validate, trace 0.2us --- 30 GB = ~60 minutes   --- 500 cts per segments of the rear layer