Engineer access to the system from LabWindows

Tunneling to Digital Gammasphere – crib notes

This document will summarize how to remotely access the 'engineering PC' in the data room at Gammasphere to allow diagnosis of issues from remote machines. This PC runs Windows and has on it a program called 'GammaWare'. GammaWare is a LabWindows (copyright of National Instruments, see http://www.ni.com/lwcvi/) program that provides low-level register/bit access.

General Notes

1. The subnet for all machines in the data room of Gammasphere is “onenet”. To get to “onenet” you must have an X windows package running on your Windows machine and then connect to sonata.phy.anl.gov.
2. For the X windows package, I’m currently using Cygwin. Cygwin is huge but it works. You can use Putty plus some other X windows package if you like.

Connecting to Sonata

1. Once you’ve got an X windows package running on your machine, the Unix/Linux command to connect is ssh –Y sonata.phy.anl.gov. You have to use the –Y to pass graphics.
2. When I connect with Cygwin it passes my login name from the PC to sonata.
   1. You may get a couple of notes/errors/warnings about not being able to create /home/xxx/.ssh. Big deal, who cares, ignore.
   2. You may also get an error about not being able to establish the authenticity of sonata.phy.anl.gov, so answer ‘yes’ to continue anyway.
3. You should then see the ANL/DOE splash about security and then must enter your ANL domain password.
4. Another warning about ‘No xauth data; using fake authentication data for X11 forwarding’ may occur that is also safe to ignore.
5. After those warnings and nags you should have a Linux prompt. There are no files to find and what few directories are present you don’t have permission to explore. All you can do from here is SSH to somewhere else. There are three places of interest to go:
   1. The PC at Gammasphere
   2. The DGS linux machine
   3. The DSSD linux machine

Accessing the Windows PC at Gammasphere

Assuming you have connected to sonata, to get to the PC at Gammasphere do the following:

1. Enter the command rdesktop wdgs –g 1280x1024. This will pop up a Windows login screen. The machine’s name is wdgs; the –g 1280x1024 sets the aspect ratio and size of the viewing area.
2. Enter the user name topoadmin. This used to be the topological trigger test stand machine, so we kept the username.
3. The password is the “standard Gammasphere password” used on all the onenet machines. All you leftover hippies from the 60s will remember it easily.
4. Once you’ve connected to the PC, you may then run the LabWindows program to access Digital Gammasphere’s trigger system. For example, here’s a screenshot.

"WDGS" is a little Windows laptop that will usually be found on top of some relay rack. It is an ANL laptop that John Anderson used from 2010-2017. It runs Windows 7. WDGS has tools such as IMPACT, Chipscope and Xilinx ISE 14.7 along with other basic stuff. The purpose of WDGS is to provide JTAG and/or GammaWare access into working systems for diagnosis.

Connecting to Digital Gammasphere and DFMA

Starting from Sonata again, enter ssh –Y dgs@dgs1 to connect to the DGS Linux box. You will be prompted for a password, the “standard Gammasphere password” is required. This is a full Linux machine, all the standard commands are available. Various people store files here, I recommend leaving everything alone. A similar technique connects you to DSSD/DFMA. Once connected to Sonata, enter ssh –Y dgs@nat2 to connect to that system. The same ‘standard Gammasphere password’ is used.

Using the EPICS systems directly

EPICS is the underlying means of communication to all the boards in the system, whether you're using GammaWare or running a user interface from a Linux box. Every register is mapped to a process variable, or 'PV'. While it is possible from the Linux prompt to access these PVs directly through "put" and "get" commands, this is normally done through a graphical user interface. Historically there was an interface develope for GRETINA written by Carl Lionberger using EDM. We call this 'Carlware'.

 To access the “Carlware” interface to Digital Gammasphere, enter the command dgscommander.  A couple of EDM windows will pop up, the main one looks like this:

Once on machine dgs@darek, the command to start up that version of ‘Carlware’ is dfmacommander. The main window there is essentially identical to that of DGS, save that the title will be FMA-ddaq Main Controller instead of DGS Main Controller. However, some small other differences become visible when you start clicking things; the FMA-ddaq version has 5 Routers listed under the “trigger’ button but DGS only has 3.

The CSS Interfaces and use of Excel register maps

Tim Madden has implemented different GUI interfaces using the Control System Studio program. They functionally perform the same things that Carlware does - acting as a graphical front end to the EPICS process variables. However, Tim has developed scripts that allow register maps for firmware, documented in Excel, to regenerate not only the EPICS PV database but also the CSS gui. This is a great way to update the systems when new firmware is released.

Using ‘Carlware’

‘Carlware’ is a fairly obvious GUI although it does have way too many subsidiary windows. For example, if you want to check the state of the transmit enable register of Router 2 in the trigger system, you have to

• Click the ‘trigger’ button and select router 2; this causes a window to pop up about that Router.
• Click the ‘serdes’ button on that screen.
  • That pops up another window with SERDES info.
• Click the ‘Link Controls’ button and select ‘Transmit Pwr’.
  • A third window pops up for just that register, with buttons for each bit.

Thankfully, Mike Carpenter has made a couple of scripts that do much of the work for standard initialization.

Specific ‘Carlware’ issues

1. Many registers of the trigger, including all the diagnostic FIFOs, are not accessible from ‘Carlware’.
2. There is no generic VME “peek/poke” method that allows access to registers not defined in the interface.
3. We’re still finding small bugs in the interface.

Changes to ‘Carlware’ are still occurring at LBNL for the GRETINA system, but their versions and versions found here at DGS and DFMA have differences and continue to diverge.

For those not familiar with EPICS, recall that global process variables have to be hosted somewhere. In the DGS and DFMA implementations of EPICS, this is done on a Soft IOC running on the Linux machine. If the Soft IOC isn't running, many features will not work.

VME Peek and Poke using 'Carlware'

It is possible to directly access registers within the VME modules by use of the console interface of each processor board (called an “IOC”, for Input/Output Controller). Clicking on the ‘Terminals’ button gives access to the different terminals. The ‘Soft IOC’ is a virtual IOC emulated in software; the others are all ‘real’ processors. From the console port various commands are available; the only three of interest to most people are

• ‘d’, the command that displays memory locations,
• ‘m’, the command that modifies memory locations, and
• ‘reboot’, to soft-boot the processor.

Since VME addresses are memory-mapped within the IOC processor, judicious use of the ‘d’ and ‘m’ commands provides a form of peek-and-poke access to registers within the module. Examples of using them are shown below.

Peeking into registers

To ‘peek’ we use the ‘d’ command. The proper format of ‘d’ is d <addr>,<nunits>,<wordwidth> where <addr> is the address, in processor memory space, where you want to start the display. <nunits> is the number of words to display and <wordwidth> is the size, in bytes, of each word. By default, VxWorks sets the base address of the VME window to 0xe8000000. Then you have to know which board in the system you want to talk to, because triggers and digitizers use geographic addressing, and then you need to know what register to play with.

Generally, the DGS and DSSD systems are set up so that the IOC is in slot 0 (leftmost) and the boards are located in slots 2,3,4,5. The DSSD trigger crate, though, is completely stuffed with the IOC in slot 0 and triggers in slots 1,2,3,4,5 and 6. By convention, the Master is the left-most trigger, but that is not a requirement. The slot number is the third hex digit in the address. So, a correctly formed read of the read-only registers of the Router in slot 2 of the DSSD crate would be d 0xe8200100,64,4 as the 0xe820 means ‘slot 2’, the following 0100 means ‘starting at address 0x100 within slot 2’, the 64 means read 64 registers and the 4 means the registers are accessed as 4-byte (32 bit) elements. The result of this (in the DSSD system) is

And a quick sanity check shows that register 0x0158 (CODE_DATE) is the expected 0x0410 (last code revision is dated April 10th) and that register 0x015C (CODE_REVISION) is the expected 0x21FF (board pcb revision 2, board type 1 (router), code revision FF (experimental)). This read also shows that the SERDES links A-H have had 1 or 2 lock losses by the values in 0x01B0 – 0x01CC. A similar read of the Router in slot 5 shows that two of the digitizers are misbehaving as shown by the large counts in 0x01C4 and 0x01C8.

Conversely, writing to a register is also possible. While not generally recommended for anyone except a complete expert, the technique is to use the ‘m’ command. The form is m <address>,<width> and it allows you to go through multiple locations. ‘m’ displays the current value and assumes you are entering a new value. Simply hitting ‘enter’ skips to the next location leaving the current one unchanged. You have to type a ‘q’ and hit ‘enter’ to exit the ‘m’ mode. It is critical to use the <width> parameter correctly or garbage gets written into the module. For DGS/DSSD modules, <width> must be set to 4.