View source for Known problems and (Hopefully!) solutions

== List of possible symptoms ==

- VME buffer number goes to 0 and remains there -> [solution 1, solution 2]

- Rate windows, and possibly other windows too, are unresponsive (usually grey out) [solution 1]

- Time stamps are out of lock [solution 3]

- Trace window will not update [solution 4]

- Following error comes up on after IOC boot -> No known solution!
 numMonitoredChans 5 firstMonitorCount 1 assignCount 7 firstConnectCount 3
 numMonitoredChans 5 firstMonitorCount 1 assignCount 7 firstConnectCount 3
 numMonitoredChans 5 firstMonitorCount 1 assignCount 7 firstConnectCount 3
 numMonitoredChans 5 firstMonitorCount 1 assignCount 7 firstConnectCount 3
 numMonitoredChans 31 firstMonitorCount 23 assignCount 39 firstConnectCount 31
 numMonitoredChans 31 firstMonitorCount 23 assignCount 39 firstConnectCount 31
 numMonitoredChans 31 firstMonitorCount 23 assignCount 39 firstConnectCount 31
 numMonitoredChans 31 firstMonitorCount 23 assignCount 39 firstConnectCount 31
 numMonitoredChans 31 firstMonitorCount 23 assignCount 39 firstConnectCount 31
 numMonitoredChans 31 firstMonitorCount 23 assignCount 39 firstConnectCount 31
 numMonitoredChans 31 firstMonitorCount 23 assignCount 39 firstConnectCount 31
 numMonitoredChans 31 firstMonitorCount 23 assignCount 39 firstConnectCount 31

== Solutions ==

1) Input/Output Controller (IOC) has lost contact with the system. Solutions include, i) "soft" reboot of individual IOC, ii) "hard" reboot of IOC, or iii) restart of entire system. Try them in the corresponding order written above.

------- Update by Ryan Nov-5-2019
With database, the buffer will be monitored every 3 sec. When buffer below 300, the script ( Database.py ) will stop the ACQ and pause for 30 sec, and then will switch on the ACQ again like nothing happened.

== JTA 20150727 ==
3) If timestamps are out of lock the typical reasons are
* The digitizers are running from the internal clock and are not listening to the trigger.
* Issue an Imperative Sync from the master trigger.
* Verify that the digitizer's SERDES is locked to the trigger system.
** Verify that the digitizer is set to be using the clock from the SERDES, not the internal clock.
* Verify that the trigger system is sending timestamps.
** There is a tree of dependence here. 
*** First the master trigger has to be set up to be sending synchronization patterns out all the SERDES links and its drivers need to be on.
*** then the router triggers must be set up to be sending synchronization patterns back up SERDES link L to the master and the drivers need to be on.
*** then the master trigger LINK_INIT state machine can be released, should ask for an ACK, an be given one.
*** then the routers must be set to use the clock from the SERDES, not the local oscillator.
*** At this point the digitizers can drive SYNC to the router and the router LINK_INIT machines can be set up.
*** Finally, then, the digitizers can be switched to use the SERDES clock from the router.

There should normally be a bash script that does all this for you.  The GUI screen for the trigger system should indicate the lock status of the triggers and allow you to issue the Imperative Sync.  The trigger summary window from DGS is a good example of how to run things and should be emulated.