Sorting Data

From HELIOS Digital DAQ
Jump to navigation Jump to search


This is a guide to help sort large amounts of DigiOS data using the GEBMerge and GEBSort_nogeb procedures.

  • GEBMerge is used to time-stamp sort all digitizer / ioc data files of type gtd
  • EventBuilder is used to create events in the data and process the data if desired. Presently, it passes all of the header information to a ROOT TTree file. Trace data can also be passed if so chosen.
  • EventBuilder_trace is same as GEBSort_nogeb, but with trace

Both files have *.chat files that define the parameters of the sort. For example, the number of pieces of data to sort, or the coincidence window defining an event.

LCRC Computing

i) Request account ii) get SSH key iii) login to blues.lcrc -> on the login node iv) setup "soft" environment (load software to evn) v) compile code in home

Useful commands

[crhoffman@blogin1 rcnp]$ lcrc-qbank -q balance
Project                   Balance (corehours)
------------------------- -------------------
*HELIOS                                 50000
startup-crhoffman                        8000

For example, to change your default project from startup-joeuser to anl-mcs-support, you would type the following command on any of the login nodes:

[joeuser]$ lcrc-qbank -s default anl-mcs-support
Previous Default Project: startup-joeuser
New Default Project: anl-mcs-support

transfer data

- using globusconnectpersonal
- create globus account online - w/ ANL
- run globusconnectpersonal locally
- setting up for Sonata 022a3522-00d9-4fee-9235-7fe797e56736
- downloaded on Sonata - requires Tcl/Tk
- from online globus point and login to LCRC
- can setup script on 'local' machine to 'push' data to LCRC
- go to transfer after logging into
- Find "local connect" then choose LCRC DTN
- "local connects" on both DigiOS [DigiOS DAQ] and Sonata
- select proper directories (may have to login to LCRC)
- click options for transfer, click arrow to start transfer

sorting data on LCRC

- using the Proof with root selectors, e.g., rcnp data
- how to run interactive job

If you need to use a remote X11 display from within your job , add the -v DISPLAY option to qsub as well (this is for 1 minute of time): $ qsub -I -l walltime=00:01:00 -v DISPLAY waiting for job to start job ready To quit your interactive job, run: $ logout

Submitting a bash script with qsub and the PBS example script that I got working (rcnp_selector.pbs) or link to better description (File:PBS Script 0.pdf):


#PBS -N rcnp
#PBS -l nodes=1:ppn=16
#PBS -l walltime=0:05:00

cd /home/crhoffman/rcnp_benchmark/rcnp
root -l -b /home/crhoffman/rcnp_benchmark/rcnp/RunSelector_LCRC.C

the run

qsub rcnp_selector.pbs

submits job in the que, gives it an id number as well, must wait for it to run, various commands to check on it
showq [-i] [-u uname]

when finished, get two files in home directory

rcnp.e1664448  rcnp.o1664448

to do ... setting up multiple instances of same process 'swift multiple run numbers at once

to do...

Transferring Data

When a computer is directly connected to the LAN of NAT Box, simply use the rsync command with the ip address of the digios computer

rsync -avh --progress dgs@* .

This reached ~100 MB/s with computer directly connected to either the NAT Box or the switch

Trying to setup a globusconnect point on the DigiOS DAQ computer directly



To run the globusconnect and to connect on DigiOS, then need to login to the LCRC to be able to transfer the data


  • Go to /music/helios2/DigiOS/working
  • Check that data file(s) in data directory
ls -ltr ../data
-r--r--r-- 1 crhoffman helios 21250344 Oct  6 11:12 data_run_004.gtd_000_0103
-r--r--r-- 1 crhoffman helios 1801997700 Oct  6 11:15 data_run_004.gtd_000_0102
  • Merge run 004
./gebmerge 004
RUN 004: GEBMerge started at Tue Oct 11 16:07:03 CDT 2016
RUN 004: GEBMerge DONE at Tue Oct 11 16:07:40 CDT 2016

If you get an error about no GEBMerge, go to GEBSort directory and type

make GEBMerge
  • Next run GEBSort_nogeb to generate root file
./gebsort 004
GEBSort started sorting run 004 at Tue Oct 11 16:11:16 CDT 2016
from GEBSort.cxx on Oct  7 2016 15:45:02
...tons of other crap...
done saving rootfile "/music/helios2/DigiOS/root_data/run004.root
read 611072 events; Pars.beta=0.0000; time since last update 0 minutes

sorted timestamp range 594573853956-->7526345374265: 6931771520309
that is 69317.7 seconds or 19h15m17s
^^^^^ any timestamp jumps will upset this accounting

hit statistics per type
14 GEB_TYPE_DGS             1957499 ;     28.24 Hz 
read a total of              1957499 ; header/payloads

Duration of sort in real time 45.02 s
Sort time ratio (real t / dTS) SMALLER IS BETTER:    0.00065

boniva sancta! ...GEBSort (unexpectedly) did not crash!

** GEBSort is done at Tue Oct 11 16:12:03 2016

GEBSort DONE at Tue Oct 11 16:12:03 CDT 2016
  • Now you can do what you want with the root file, (note: may have to run ssetup root_v5.32.00 for root to work), e.g.,
root -l ../root_data/run004.root 

Will run a little selector to generate E vs Z [figure below], noting that in this case E was determined from the difference between the pre / post rise sums as determined by the firmware, e.g., not using the trace data.

Energy vs. Z as generated by scripts in code directory using rootfile from GEBSort_nogeb.

Locations of Working Sort Codes

As of now, I (CRH) have been using a set of codes on malaguena.onenet. I login using my own username (not helios@phy), and I reccommend this for everyone. If we need to change some permissions, or get people on the same group we can do it. But it will be confusing to have everyone as helios.

The current working directory for the GEBMerge and GEBSort_nogeb codes is /music/helios2/DigiOS/GEBSort

Other directories in music/helios2/DigiOS include

codes  data  GEBSort  merged_data  root_data  working

As you will see below, running GEBMerge and GEBSort_nogeb can be involved, hence, in the working directory, there are a few *.sh scripts to help run them, e.g., to merge run 004 in the data directory, in working type 004

Please note that the file locations are hardcoded at this point so change them if you intend to play around with things!

Running GEBMerge

It is run in the following way (NOTE: A map.dat file is required in execution directory even if it is empty??) :

GEBMerge merged_output_file.gtd data_file(s).gtd

The includes a number of parameters defining the sort. My (CHR) current working on is in /music/helios2/DigiOS/working

Also, a more simple way to run the merge is to use the script as 004

to sort the data run 004 in the data directory

Runing GEBSort_nogeb

Please note, if you modify GEBSort, you want to make GEBSort_nogeb (NO Global Event Builder), otherwise it will fail because it is meant to get realtime data from Gretina (i think).

GEBSort_nogeb is run by

GEBSort_nogeb -input input_type [input_data_file.gtd] -rootfile
output_root_file.root rootfile_option -chat
  • input_type -> disk : to indicate from file (not online)
  • input_data_file.gtd : can be merged or single-IOC file (I believe)
  • rootfile_option -> RECREATE or UPDATE : whether to create new or add to the root file specified

Other files required by GEBSort_nogeb (all to be placed in working directory)

  • map.dat : gives ID’s and types of channels (can be an empty file, but must be there it seems)

GEBSort_nogeb OUTPUT

The present version of the code uses the bin_rcnp.* files to generate a ROOT TTree with the information from the header. There are still some issues with the full readout of the header, but all information for timestamps and energy is there now (extra stuff is for better P/Z corrections, etc.). The root TTree should be nearly identical to the parameters given in the typedef struct DGSEVENT' which is found in GTMerge.h