Linking Systems Together

Linking Multiple Systems Together

    The trigger logic implemented for the various DAQ systems and the hardware design of the trigger module allows for the possibility to link multiple systems together.  This allows both clock synchronization and trigger sharing to occur.  A number of rules must be obeyed, so read carefully.

Clock Sharing

    The basic method of clock sharing that most users are familiar with is that the master trigger sends clock to the routers, and the routers re-distribute that clock to the digitizers.  Many users also understand that this tree of clock distribution has been extended to include Digital Gammasphere (DGS) sending the clock to DFMA, as shown here.

     Those that have been involved from the beginning will recall that the DGS master trigger could also be connected to 'Analog' Gammasphere using the GITMO module - and use the clock from 'Analog' Gammasphere.  Those that have been involved with hosting external detectors such as CHICO, Microball, ORRUBA, etc. also understand that there is clock sharing associated with the MyRIAD module.  A more full picture of this clock distribution hierarchy is shown here.

The rules of the game as currently understood by users

     So people know that you can hook a MyRIAD to DGS, and you can hook DFMA to DGS, and that with appropriate software incantations the clock and timestamp can be distributed from DGS to the other two systems.  That is correct, and that is true, but it is not the whole and complete truth of how the firmware works.

How clock sharing REALLY works

   How the clock sharing works in full detail is as follows.

• Any master trigger can be declared as the "timing monarch" that is the source of the clock.
• Which master trigger is the "timing monarch" is defined solely by cabling.
  • If a master trigger has a cable connected to it's link L then that master trigger can be told to be subservient to whatever is on the other end of the link L cable.
  • The hierarchy is defined by what cables plug into link L of what boards.
  • This is why all the master-to-router cables go from any link of the master always to link L of the router.
• There is a limit as to how far the hierarchy can descend.
  • Each time the clock is sent out of a board and received by another, clock jitter accumulates with each 'hop'.
  • In the DGS-DFMA setup that people are used to, there are six 'hops' to get a trigger from DFMA back to DGS when DGS is the clock source :
    • DGS to DFMA master -> DFMA master-router -> DFMA router-digitizer -> DFMA digitizer-router -> DFMA router-master -> DFMA master to DGS
    • Measurements show that the accumulated jitter per 'hop' is about 35ps.
    • So in the DGS-DFMA setup there would be a total accumulated jitter of (6*35), or 360ps.
    • The serial data stream is 1Gbps, or 1ns (1000ps) per bit.
    • If the ratio of the accumulated jitter divided by the bit period is greater than 50%, the link WILL HAVE BIT ERRORS.
  • Thus the number of 'hops' in the current system is at a maximum and hanging a 3rd system behind DFMA will not work.

Any master trigger starts up using its own internal clock. Then, by writing a bit to a register, a master trigger agrees to use the clock that comes in on link L. The firmware has an automatic fallback provision that will cause any master trigger to fall back to using the internal clock irrespective of the software selection if there is no LOCK indication from the SERDES chip of link L. Thus it is necessary to establish the SERDES link between two master triggers and verify that link L is locked BEFORE telling the "subservient" master trigger to use the clock from link L.

Understanding Link usage

   Each master trigger board has three "special" SERDES links named "L", "R" and "U".  Originally this was meant to be read as "Left", "Right" and "Up" for drawing complicated link connections, but the convention of direction was abandoned unused.  These three links are "special" in the sense that they are intended to be used as system interconnect links as opposed to the internal connections of links A through H.  More precisely, what this means is that the data formats that these links send and receive are slightly different than the data format used to connect with routers.  Each of these three links has a unique purpose and unique characteristics.

• Link L is unique in that it alone can receive a clock from an external source that can be used in place of the board's internal clock.
  • Link L can receive two different data formats, either the GITMO format or the format from another master trigger.
• Link R is intended to receive data from another master trigger.
• Link U is historically used with the MyRIAD module to receive trigger information from an external detector.
  • Link U can be programmed to receive data from another master trigger.

The concept of Propagation Control

Every master trigger is the master of its own detector. However, each master trigger that is receiving data from another master trigger from the SERDES links (L, R or U) can selectively propagate information received from another master trigger into itself. There is a Propagation Control register associated with each of the three links that is bit-wise mapped to enumerate what information is allowed to enter. This is described in the Trigger Timing and Control Link Specification document that I know you haven't read, so here's the text again.

These Propagation Control registers are important not only to set up the timestamp, but also in the setting up of shared triggers across systems.

Timestamp sharing and Imperative Sync

    There is a difference between sharing the CLOCK and sharing the TIMESTAMP.  The timestamp logic of the subservient master trigger has a separate enable bit to allow the subservient master to respond to Imperative Sync commands sent from the "timing monarch".  The precise rule here is:

• If the LINK L PROPAGATION CONTROL register, bit 0, is SET, and the subservient master has set its link L to be receiving commands from another master trigger, then the subservient master will respond only to Imperative Sync from the "timing monarch" AND cannot issue its own local Imperative Sync.
• If either of those two conditions are not met, then the subservient master, even if using the clock from the timing monarch, will IGNORE any Imperative Sync from the timing monarch but CAN issue its own Imperative Sync within its own system.

Collection of triggers from one master trigger to another

   Each master trigger receives copies of all the 'local' (within a given detector) trigger accept messages from all the other master triggers it is connected to through links L, R and U.  Hypothesize that the setup in play is Digital Gammasphere (DGS) as the "timing monarch" with DFMA and X-Array as the "subservient" master triggers.  Just to fill out the example, also assume that there's some other external detector in the setup that is using a MyRIAD.  Further assume that the wiring setup is as follows:

• DGS link R is connected to DFMA, link L.
• DGS link L is connected to X-Array, link L.
• DGS link U is connected to the MyRIAD in the external detector.

This is a legitimate setup because link L can be configured by writing to DGS's registers to use 'remote master' format, link R is natively 'remote master' format, and link U of DGS can be configured by writing to registers to use 'MyRIAD' format. This setup then means that the DGS master trigger has to somehow attempt to combine information not only from itself but from the other setups to make multi-system trigger accept messages. The overall data flow by which this occurs is shown in the following picture.