23. Cabling & software setup [/]
Appendix extended
This appendix gives a short overview on how the cabling of the detector needs to be set up and how to set up a computer to flash the firmware TOF. Finally, an example is given on how to start up TOS and start a background run. These are mainly notes taken for myself, but I think they should be useful for anyone attempting to run this detector.
23.1. Virtex V6 cabling
The following pieces are required:
- power supply
- 2 HDMI to intermediate board
- mini USB into JTAG port on backside (to computer)
- RJ45 from ethernet port into 2nd ethernet card on DAQ PC
The RJ45 connection is only required to flash the firmware onto the Virtex.
23.2. Detector cabling
The following is the cabling for the FADC and the scintillators to the intermediate board. It's a useful reference when connecting the cabling!
FADC Trig out (on the FADC):
- trig out -> level adapter (NIM module) into NIM IN, set to
NORM/ COMPL (this is active) TTL out -> TTL signal clipper (TTL 5V -> 2.4V) into port markedI
TTL signal clipper -> adapter board into left LEMO at back (viewed from behind the crate; cable still at CAST)
FPGA to FADC (shutter signal)
- Adapter board right LEMO at back (viewed from behind crate) ->
level adapter (NIM module), set to NORM /
COMPLinto TTL IN NIM out -> FADC EXT EN TRIG
Veto scintillator (NOTE this may be WRONG; input to adapter board that is); signal order is reversed!
- Adapter board left LEMO on top (viewed from behind the crate, input number 1 on adapter board) -> discriminator OUT (top discriminator in NIM module) discriminator IN -> back of Amplifier Discriminator NIM module amplifier input -> veto scintillator signal The signal is needs to be 2.4 V TTL signal!
23.3. Vivado / ISE on void linux & flashing Virtex V6
The setup described here was written for Void Linux, but of course should hold generally with minor modification.
Installing Vivado on void linux is relatively straight forward (does not mean it's not annoying).
First of all note that apparently Vivado version 2020.3 dropped support for many devices! That's why I decided to download the 2020.2 self installing linux web installer from here: https://www.xilinx.com/member/forms/download/xef.html?filename=Xilinx_Unified_2020.2_1118_1232_Lin64.bin (need a Xilinx account of course).
Installation of it is simply done by running the installer and waiting
a long time (it seems to work just fine on Void). On tpc19
I
installed it under /data/Xilinx
(that's the 2 TB HDD in there).
It complained at the end about some Python runtime error setting up some stuff or something, but said compilation was otherwise successful.
Note: It needs about 56 GB (!!!) of space!
After installation we need to do 3 more things to actually run it:
install (if not already done)
libtinfo
:sudo xbps-install -S ncurses-libtinfo-libs ncurses-libtinfo-devel
However, this only installs version 6 of
libtinfo
, but we need 5. Fortunately, 5 and 6 are API compatible, so we can link 6 to version 5:cd /lib64 sudo ln -s libtinfo.so.6 libtinfo.so.5
we need to source the
settings64.sh
file here:source /data/Xilinx/Vivado/2020.2/settings64.sh
and finally we need to set some Java related environment variable, because otherwise under a tiling WM we only get a white screen upon launch:
export _JAVA_AWT_WM_NONREPARENTIN=1
After that it should launch succesfully.
23.3.1. Installing ISE
ISE has the advantage over Vivado that it allows for a so called "lab tools" installation, which only weighs about 5.5 GB (compared to Vivado's > 50 GB.
Download the installer from: https://www.xilinx.com/member/forms/download/xef.html?filename=Xilinx_ISE_DS_Lin_14.7_1015_1.tar (requires Xilinx account).
Untar the file somewhere and execute the xsetup
with superuser
access. This is in case the chosen installation path is not user writable or
the user wishes to install the USB drivers. The latter is very useful,
but possibly does not work. It's relatively straightforward to install
USB drivers manually though (see below 23.3.2).
tar xf Xilinx_ISE_DS_Lin_14.7_1015_1.tar
cd Xilinx_ISE_DS_Lin_14.7_1015_1
sudo ./xsetup
Simply follow the installation instructions and select the "LabTools"
package. It contains impact
, which is all we care about to flash the
Virtex V6.
To run impact
we need to source the settings64.sh
file:
source <ISE_PATH>/settings64.sh
23.3.2. Install USB drivers
To see whether the correct USB drivers are installed on a machine, connect the Virtex V6 board using a mini USB cable in the JTAG port to the computer. In the terminal:
lsusb
we should see a Xilinx device. The Bus and device number of course depends on the machine and port. If the device is recognized, the ID can either be:
03fd:000d Xilinx, Inc.
or
03fd:0008 Xilinx, Inc. Platform Cable USB II
The former means only some generic Xilinx driver was loaded. The latter is the one we want.
There are 4 different ways to install USB drivers, 3 are very similar to one another:
Using the USB drivers shipped with Vivado. These are found in:
<Vivado_Path>/data/xicom/cable_drivers/lin64/install_script/install_drivers
From here in theory we should just have to run the
install_drivers
script with sudo rights and they should be installed. This does not work on Void Linux. Ref: https://www.xilinx.com/support/answers/59128.htmlUsing the USB drivers shipped with ISE. These are found in:
<ISE_Path>/LabTools/LabTools/bin/lin64/install_script/install_drivers
Here we also find a
readme.txt
, which contains instructions for the installation. In theory we should just have to run theinstall_drivers
script with sudo rights and they should be installed. This also does not work on Void Linux. Ref: https://www.xilinx.com/support/answers/54381.htmlUsing the USB drivers downloaded manually from Xilinx: Download the files from here: https://secure.xilinx.com/webreg/clickthrough.do?cid=103670 which gives us a
install_drivers.tar.gz
. Untar:tar xzf install_drivers.tar.gz cd install_drivers
and again, in theory we should be able to run the
install_drivers
script here. On Void Linux with this approach I got a bit farther. I had to installfxload
, which I used from here: https://github.com/esden/fxload, compiled it and symlinkedfxload
to/sbin/fxload
. That made the installer (which again has to be run with root) find fxload. However, from there many more errors happened, apparently because the scripts use/bin/sh
, but expects a real bourne shell (bash) and not dash (typically used in ubuntu and many other distributions). I patched all occurences of/bin/sh
to use explicitly/bin/bash
, which got me farther, but in the end I still could not successfully install the drivers. Ref: https://www.xilinx.com/support/documentation/user_guides/ug344.pdfThe best way (and simplest) is to follow the Arch Wiki: Ref: https://wiki.archlinux.org/title/Xilinx_ISE_WebPACK#Xilinx_Platform_Cable_USB-JTAG_Drivers The instructions here make use of a custom USB driver package by "zerfleddert". Essentially just clone the repository:
git clone git://git.zerfleddert.de/usb-driver
and call compile by running
make
. Afterwards we need to run thesetup_pcusb
script with the path to the ISE installation (again with superuser rights). However, the path to the ISE installation is not just the path to theISE
directory, but:sudo ./setup_pcusb <ISE_PATH>/LabTools/common
(the
common
directory is the one that contains the actual project). From here I compared my (now created)/etc/udev/rules.d/xusbdfwu.rules
file with the file shown in the Arch wiki. They are not exactly identical, but I tried it first before modifying it. To be certain I reloaded the udev rules anyway:sudo udevadm control --reload-rules
Afterwards, reconnecting the Virtex V6 board and running
lsusb
showed the correct driver. Flashing the firmware works correctly both in Vivado as well as in Impact now.
23.3.3. Setting up ethernet device
23.3.4. Ethernet connection with Virtex
The ethernet connection with the Virtex needs to be set up manually.
On the one hand it is required to use a static IP address for the
secondary ethernet device and in addition we need to set an ARP entry
(note: the arp
program is part of the net-tools
package, name same
in ubuntu & void linux).
Under Ubuntu said setup can be done using the network manager. In Void
we need to use ip
from the terminal.
The settings are as follows:
- IP address: 10.1.2.3
- Subnet: 24
Setup of the device can be done according to the example here: https://docs.voidlinux.org/config/network/index.html namely (with superuser rights):
ip addr show # check the name of the correct, secondary device ip link set dev enp4s0 up # name on tpc19 ip addr add 10.1.2.3/24 brd + dev enp4s0
afterwards we can set the ARP entries (ref: https://confluence.team.uni-bonn.de/display/PHYGASDET/How+to+automatically+set+ARP+entries):
arp -i enp4s0 -s 10.1.2.2 AA:BA:DD:EC:AD:E2
23.3.5. Make these two steps automatic under void
In principle it should be enough to set the above steps into /etc/rc.local
.
23.3.6. Temperature readout via MCP2210 and MAX31685
Ref: https://datasheets.maximintegrated.com/en/ds/MAX31865.pdf
The temperature readout via the MCP2210 micro-controllers installed on the intermediate board is done via SPI through a micro USB port on the intermediate board. This micro-controller talks to a MAX31685 for each of the sensors.
The micro-controller is powered via an additional power line! That means one needs to use one of the "big" power supplies with an additional +5V input.
Using such a power supply and plugging the micro USB port from the
intermediate board into the PC, should yield an entry like the
following on lsusb
:
lsusb
Bus 001 Device 024: ID 04d8:00de Microchip Technology, Inc. MCP2210 USB to SPI Master
The MCP2210 communication is done through code built on an open source library by Kerry Wong: https://github.com/kerrydwong/MCP2210-Library
In TOS this is embedded using the HID API.
To get the communication working, the final step is the udev
rules. The file
https://github.com/kerrydwong/MCP2210-Library/blob/master/99-hid.rules
needs to be placed in /etc/udev/rules.d/
.
Since we use the HID API the second entry is the relevant one. The
file as it is being used on tpc19
at the moment.
/etc/udev/rules.d/99-mcp2210.rules
:
# This is a sample udev file for HIDAPI devices which changes the permissions # to 0666 (world readable/writable) for a specified device on Linux systems. # If you are using the hidraw implementation, then do something like the # following, substituting the VID and PID with your device. Busnum 1 is USB. # HIDAPI/hidraw KERNEL=="hidraw*", ATTRS{busnum}=="1", ATTRS{idVendor}=="04d8", ATTRS{idProduct}=="00de", MODE="0666" # Once done, optionally rename this file for your device, and drop it into # /etc/udev/rules.d and unplug and re-plug your device. This is all that is # necessary to see the new permissions. Udev does not have to be restarted. # Note that the hexadecimal values for VID and PID are case sensitive and # must be lower case. # If you think permissions of 0666 are too loose, then see: # http://reactivated.net/writing_udev_rules.html for more information on finer # grained permission setting. For example, it might be sufficient to just # set the group or user owner for specific devices (for example the plugdev # group on some systems).
After writing this file (take care to check that the USB device is
actually on USB bus number 1, as in the case of the example output of
lsusb
above), we can reload the udev
rues:
sudo udevadm control --reload-rules
and then replug the USB connection to the intermediate board.
Now, TOS (or a standalone program using the MCP2210) should work fine.
23.4. Setting up the chips in TOS
This section is specific to the Septemboard used at CAST.
Following the steps described in the shifter documentation ./../org/Doc/ShiftDocumentation/shifter_documentation.html
#+BEGIN_SRC python 7 # number of chips 4 # preload SetChipIDOffset 190 lf # 7 times enter to load default paths uma 1 # Matrix settings 0 1 1 0 LoadThreshold # load threshold equalisation files 4 # write matrix 3 # read out 3 ActivateHFM SetFadcSettings Run 1 # run time via # frames 0 0 0 2 # shutter range select 30 # shutter time select 0 # zero suppression 1 # FADC usage 0 # accept FADC settings
The above would launch a full background run.