Linking Systems Together

Linking Multiple Systems Together

    The trigger logic implemented for the various DAQ systems and the hardware design of the trigger module allows for the possibility to link multiple systems together.  This allows both clock synchronization and trigger sharing to occur.  A number of rules must be obeyed, so read carefully.

Clock Sharing

    The basic method of clock sharing that most users are familiar with is that the master trigger sends clock to the routers, and the routers re-distribute that clock to the digitizers.  Many users also understand that this tree of clock distribution has been extended to include Digital Gammasphere (DGS) sending the clock to DFMA, as shown here.

     Those that have been involved from the beginning will recall that the DGS master trigger could also be connected to 'Analog' Gammasphere using the GITMO module - and use the clock from 'Analog' Gammasphere.  Those that have been involved with hosting external detectors such as CHICO, Microball, ORRUBA, etc. also understand that there is clock sharing associated with the MyRIAD module.  A more full picture of this clock distribution hierarchy is shown here.

The rules of the game as currently understood by users

     So people know that you can hook a MyRIAD to DGS, and you can hook DFMA to DGS, and that with appropriate software incantations the clock and timestamp can be distributed from DGS to the other two systems.  That is correct, and that is true, but it is not the whole and complete truth of how the firmware works.

How clock sharing REALLY works

   How the clock sharing works in full detail is as follows.

• Any master trigger can be declared as the "timing monarch" that is the source of the clock.
• Which master trigger is the "timing monarch" is defined solely by cabling.
  • If a master trigger has a cable connected to it's link L then that master trigger can be told to be subservient to whatever is on the other end of the link L cable.
  • The hierarchy is defined by what cables plug into link L of what boards.
  • This is why all the master-to-router cables go from any link of the master always to link L of the router.
• There is a limit as to how far the hierarchy can descend.
  • Each time the clock is sent out of a board and received by another, clock jitter accumulates with each 'hop'.
  • In the DGS-DFMA setup that people are used to, there are six 'hops' to get a trigger from DFMA back to DGS when DGS is the clock source :
    • DGS to DFMA master -> DFMA master-router -> DFMA router-digitizer -> DFMA digitizer-router -> DFMA router-master -> DFMA master to DGS
    • Measurements show that the accumulated jitter per 'hop' is about 35ps.
    • So in the DGS-DFMA setup there would be a total accumulated jitter of (6*35), or 360ps.
    • The serial data stream is 1Gbps, or 1ns (1000ps) per bit.
    • If the ratio of the accumulated jitter divided by the bit period is greater than 50%, the link WILL HAVE BIT ERRORS.
  • Thus the number of 'hops' in the current system is at a maximum and hanging a 3rd system behind DFMA will not work.

Any master trigger starts up using its own internal clock. Then, by writing a bit to a register, a master trigger agrees to use the clock that comes in on link L. The firmware has an automatic fallback provision that will cause any master trigger to fall back to using the internal clock irrespective of the software selection if there is no LOCK indication from the SERDES chip of link L. Thus it is necessary to establish the SERDES link between two master triggers and verify that link L is locked BEFORE telling the "subservient" master trigger to use the clock from link L.

Timestamp sharing

    There is