23. Cabling & software setup [/]   Appendix extended

This appendix gives a short overview on how the cabling of the detector needs to be set up and how to set up a computer to flash the firmware TOF. Finally, an example is given on how to start up TOS and start a background run. These are mainly notes taken for myself, but I think they should be useful for anyone attempting to run this detector.

23.1. Virtex V6 cabling

The following pieces are required:

  • power supply
  • 2 HDMI to intermediate board
  • mini USB into JTAG port on backside (to computer)
  • RJ45 from ethernet port into 2nd ethernet card on DAQ PC

The RJ45 connection is only required to flash the firmware onto the Virtex.

23.2. Detector cabling

The following is the cabling for the FADC and the scintillators to the intermediate board. It's a useful reference when connecting the cabling!

FADC Trig out (on the FADC):

  • trig out -> level adapter (NIM module) into NIM IN, set to NORM / COMPL (this is active) TTL out -> TTL signal clipper (TTL 5V -> 2.4V) into port marked I TTL signal clipper -> adapter board into left LEMO at back (viewed from behind the crate; cable still at CAST)

FPGA to FADC (shutter signal)

  • Adapter board right LEMO at back (viewed from behind crate) -> level adapter (NIM module), set to NORM / COMPL into TTL IN NIM out -> FADC EXT EN TRIG

Veto scintillator (NOTE this may be WRONG; input to adapter board that is); signal order is reversed!

  • Adapter board left LEMO on top (viewed from behind the crate, input number 1 on adapter board) -> discriminator OUT (top discriminator in NIM module) discriminator IN -> back of Amplifier Discriminator NIM module amplifier input -> veto scintillator signal The signal is needs to be 2.4 V TTL signal!

23.3. Vivado / ISE on void linux & flashing Virtex V6

The setup described here was written for Void Linux, but of course should hold generally with minor modification.

Installing Vivado on void linux is relatively straight forward (does not mean it's not annoying).

First of all note that apparently Vivado version 2020.3 dropped support for many devices! That's why I decided to download the 2020.2 self installing linux web installer from here: https://www.xilinx.com/member/forms/download/xef.html?filename=Xilinx_Unified_2020.2_1118_1232_Lin64.bin (need a Xilinx account of course).

Installation of it is simply done by running the installer and waiting a long time (it seems to work just fine on Void). On tpc19 I installed it under /data/Xilinx (that's the 2 TB HDD in there).

It complained at the end about some Python runtime error setting up some stuff or something, but said compilation was otherwise successful.

Note: It needs about 56 GB (!!!) of space!

After installation we need to do 3 more things to actually run it:

  1. install (if not already done) libtinfo:

       sudo xbps-install -S ncurses-libtinfo-libs ncurses-libtinfo-devel
    

    However, this only installs version 6 of libtinfo, but we need 5. Fortunately, 5 and 6 are API compatible, so we can link 6 to version 5:

       cd /lib64
       sudo ln -s libtinfo.so.6 libtinfo.so.5
    
  2. we need to source the settings64.sh file here:

       source /data/Xilinx/Vivado/2020.2/settings64.sh
    
  3. and finally we need to set some Java related environment variable, because otherwise under a tiling WM we only get a white screen upon launch:

       export _JAVA_AWT_WM_NONREPARENTIN=1
    

    (ref: https://forums.xilinx.com/t5/Installation-and-Licensing/Vivado-hangs-forever-with-white-window-during-startup-Linux/td-p/479058)

After that it should launch succesfully.

23.3.1. Installing ISE

ISE has the advantage over Vivado that it allows for a so called "lab tools" installation, which only weighs about 5.5 GB (compared to Vivado's > 50 GB.

Download the installer from: https://www.xilinx.com/member/forms/download/xef.html?filename=Xilinx_ISE_DS_Lin_14.7_1015_1.tar (requires Xilinx account).

Untar the file somewhere and execute the xsetup with superuser access. This is in case the chosen installation path is not user writable or the user wishes to install the USB drivers. The latter is very useful, but possibly does not work. It's relatively straightforward to install USB drivers manually though (see below 23.3.2).

tar xf Xilinx_ISE_DS_Lin_14.7_1015_1.tar
cd Xilinx_ISE_DS_Lin_14.7_1015_1
sudo ./xsetup

Simply follow the installation instructions and select the "LabTools" package. It contains impact, which is all we care about to flash the Virtex V6.

To run impact we need to source the settings64.sh file:

source <ISE_PATH>/settings64.sh

23.3.2. Install USB drivers

To see whether the correct USB drivers are installed on a machine, connect the Virtex V6 board using a mini USB cable in the JTAG port to the computer. In the terminal:

lsusb

we should see a Xilinx device. The Bus and device number of course depends on the machine and port. If the device is recognized, the ID can either be:

03fd:000d Xilinx, Inc.

or

03fd:0008 Xilinx, Inc. Platform Cable USB II

The former means only some generic Xilinx driver was loaded. The latter is the one we want.

There are 4 different ways to install USB drivers, 3 are very similar to one another:

  1. Using the USB drivers shipped with Vivado. These are found in:

       <Vivado_Path>/data/xicom/cable_drivers/lin64/install_script/install_drivers
    

    From here in theory we should just have to run the install_drivers script with sudo rights and they should be installed. This does not work on Void Linux. Ref: https://www.xilinx.com/support/answers/59128.html

  2. Using the USB drivers shipped with ISE. These are found in:

       <ISE_Path>/LabTools/LabTools/bin/lin64/install_script/install_drivers
    

    Here we also find a readme.txt, which contains instructions for the installation. In theory we should just have to run the install_drivers script with sudo rights and they should be installed. This also does not work on Void Linux. Ref: https://www.xilinx.com/support/answers/54381.html

  3. Using the USB drivers downloaded manually from Xilinx: Download the files from here: https://secure.xilinx.com/webreg/clickthrough.do?cid=103670 which gives us a install_drivers.tar.gz. Untar:

       tar xzf install_drivers.tar.gz
       cd install_drivers
    

    and again, in theory we should be able to run the install_drivers script here. On Void Linux with this approach I got a bit farther. I had to install fxload, which I used from here: https://github.com/esden/fxload, compiled it and symlinked fxload to /sbin/fxload. That made the installer (which again has to be run with root) find fxload. However, from there many more errors happened, apparently because the scripts use /bin/sh, but expects a real bourne shell (bash) and not dash (typically used in ubuntu and many other distributions). I patched all occurences of /bin/sh to use explicitly /bin/bash, which got me farther, but in the end I still could not successfully install the drivers. Ref: https://www.xilinx.com/support/documentation/user_guides/ug344.pdf

  4. The best way (and simplest) is to follow the Arch Wiki: Ref: https://wiki.archlinux.org/title/Xilinx_ISE_WebPACK#Xilinx_Platform_Cable_USB-JTAG_Drivers The instructions here make use of a custom USB driver package by "zerfleddert". Essentially just clone the repository:

       git clone git://git.zerfleddert.de/usb-driver
    

    and call compile by running make. Afterwards we need to run the setup_pcusb script with the path to the ISE installation (again with superuser rights). However, the path to the ISE installation is not just the path to the ISE directory, but:

       sudo ./setup_pcusb <ISE_PATH>/LabTools/common
    

    (the common directory is the one that contains the actual project). From here I compared my (now created) /etc/udev/rules.d/xusbdfwu.rules file with the file shown in the Arch wiki. They are not exactly identical, but I tried it first before modifying it. To be certain I reloaded the udev rules anyway:

       sudo udevadm control --reload-rules
    

    Afterwards, reconnecting the Virtex V6 board and running lsusb showed the correct driver. Flashing the firmware works correctly both in Vivado as well as in Impact now.

23.3.3. Setting up ethernet device

23.3.4. Ethernet connection with Virtex

The ethernet connection with the Virtex needs to be set up manually.

On the one hand it is required to use a static IP address for the secondary ethernet device and in addition we need to set an ARP entry (note: the arp program is part of the net-tools package, name same in ubuntu & void linux).

Under Ubuntu said setup can be done using the network manager. In Void we need to use ip from the terminal.

The settings are as follows:

  • IP address: 10.1.2.3
  • Subnet: 24

Setup of the device can be done according to the example here: https://docs.voidlinux.org/config/network/index.html namely (with superuser rights):

ip addr show
# check the name of the correct, secondary device
ip link set dev enp4s0 up # name on tpc19
ip addr add 10.1.2.3/24 brd + dev enp4s0

afterwards we can set the ARP entries (ref: https://confluence.team.uni-bonn.de/display/PHYGASDET/How+to+automatically+set+ARP+entries):

arp -i enp4s0 -s 10.1.2.2 AA:BA:DD:EC:AD:E2

23.3.5. Make these two steps automatic under void

In principle it should be enough to set the above steps into /etc/rc.local.

23.3.6. Temperature readout via MCP2210 and MAX31685

Ref: https://datasheets.maximintegrated.com/en/ds/MAX31865.pdf

The temperature readout via the MCP2210 micro-controllers installed on the intermediate board is done via SPI through a micro USB port on the intermediate board. This micro-controller talks to a MAX31685 for each of the sensors.

The micro-controller is powered via an additional power line! That means one needs to use one of the "big" power supplies with an additional +5V input.

Using such a power supply and plugging the micro USB port from the intermediate board into the PC, should yield an entry like the following on lsusb:

lsusb

Bus 001 Device 024: ID 04d8:00de Microchip Technology, Inc. MCP2210 USB to SPI Master

The MCP2210 communication is done through code built on an open source library by Kerry Wong: https://github.com/kerrydwong/MCP2210-Library

In TOS this is embedded using the HID API.

To get the communication working, the final step is the udev rules. The file https://github.com/kerrydwong/MCP2210-Library/blob/master/99-hid.rules needs to be placed in /etc/udev/rules.d/. Since we use the HID API the second entry is the relevant one. The file as it is being used on tpc19 at the moment. /etc/udev/rules.d/99-mcp2210.rules:

# This is a sample udev file for HIDAPI devices which changes the permissions
# to 0666 (world readable/writable) for a specified device on Linux systems.

# If you are using the hidraw implementation, then do something like the
# following, substituting the VID and PID with your device. Busnum 1 is USB.

# HIDAPI/hidraw
KERNEL=="hidraw*", ATTRS{busnum}=="1", ATTRS{idVendor}=="04d8", ATTRS{idProduct}=="00de", MODE="0666"

# Once done, optionally rename this file for your device, and drop it into
# /etc/udev/rules.d and unplug and re-plug your device. This is all that is
# necessary to see the new permissions. Udev does not have to be restarted.

# Note that the hexadecimal values for VID and PID are case sensitive and
# must be lower case.

# If you think permissions of 0666 are too loose, then see:
# http://reactivated.net/writing_udev_rules.html for more information on finer
# grained permission setting. For example, it might be sufficient to just
# set the group or user owner for specific devices (for example the plugdev
# group on some systems).

After writing this file (take care to check that the USB device is actually on USB bus number 1, as in the case of the example output of lsusb above), we can reload the udev rues:

sudo udevadm control --reload-rules

and then replug the USB connection to the intermediate board.

Now, TOS (or a standalone program using the MCP2210) should work fine.

23.4. Setting up the chips in TOS

This section is specific to the Septemboard used at CAST.

Following the steps described in the shifter documentation ./../org/Doc/ShiftDocumentation/shifter_documentation.html

#+BEGIN_SRC python
7 # number of chips
4 # preload
SetChipIDOffset
190
lf
# 7 times enter to load default paths
uma
1 # Matrix settings
0
1
1
0
LoadThreshold # load threshold equalisation files
4 # write matrix
3 # read out
3
ActivateHFM
SetFadcSettings
Run
1 # run time via # frames
0
0
0
2 # shutter range select
30 # shutter time select
0 # zero suppression
1 # FADC usage
0 # accept FADC settings

The above would launch a full background run.

Click on any heading marked 'extended' to open it