All programs in the TimepixAnalysis repository are command line only. While it would be quite doable to merge the different programs into a single graphical user interface (GUI), I'm personally not much of a GUI person. Each program usually has a large number of (optional) parameters. Keeping the GUI up to date with (in the past, quickly) changing features is just extra work, which I personally did not have any use for (if someone wishes to write a GUI for TimepixAnalysis, I'd be more than happy to mentor though).

Every program uses cligen ², a command line interface generator. Based on the definition of the main procedure(s) in the program, a command line interface is generated. While cligen provides extremely simplified command line argument parsing for the developer, it also gives a nice help screen for every program. For example, running the first program of the analysis pipeline raw_data_manipulation with the -h or --help option:

raw_data_manipulation -h

yields the help screen as shown in listing 16 ³. Keep this in mind, if you are unsure about how to use any of the here mentioned programs.

Further, there is a TOML configuration file in the repository (Analysis/ingrid/config.toml from the repository root), which controls many aspects of the different programs. Most of these can be overwritten by command line arguments to the appropriate programs and some also via environment variables. See the extended thesis for information about this, it is mentioned where important.

Usage:
  main [REQUIRED,optional-params]
Version: 44c0c91 built on: 2023-12-06 at 13:01:35
Options:
  -h, --help                               print this cligen-erated help

  --help-syntax                            advanced: prepend,plurals,..

  -p=, --path=        string      REQUIRED set path

  -r=, --runType=     RunTypeKind REQUIRED Select run type (Calib | Back | Xray)
                                           The following are parsed case
                                           insensetive:
                                             Calib = {"calib", "calibration", "c"}
                                             Back = {"back", "background", "b"}
                                             Xray = {"xray", "xrayfinger", "x"}

  -o=, --out=         string      ""       Filename of output file. If none
                                           given will be set to run_file.h5.

  -n, --nofadc        bool        false    Do not read FADC files.

  -i, --ignoreRunList bool        false    If set ignores the run list 2014/15
                                           to indicate using any rfOldTos run

  -c=, --config=      string      ""       Path to the configuration file to use.
                                           Default is config.toml in directory of
                                           this source file.
  ...                                         

  -t, --tpx3          bool        false    Convert data from a Timepix3 H5 file
                                           to TPA format instead of a Tpx1 run
                                           directory
  ...                                         

TimepixAnalysis mainly has a single noteworthy external dependency, namely the HDF5 (The HDF Group 1997) library. The vast majority of code (inside the repository itself and its dependencies) is pure Nim. Two optimization libraries written in C mpfit (Levenberg-Marquardt) ⁴ and NLopt ⁵ are wrapped from Nim and are further minor dependencies. Local compilation and installation of these is trivial and explained in the TimepixAnalysis README.

For those programs related to the multilayer perceptron (MLP) training or usage, PyTorch (Paszke et al. 2019) is an additional dependency via (SciNim contributors 2023). Flambeau installs a suitable PyTorch version for you.

Common other dependencies are the cairo graphics library ⁶ and a working BLAS and LAPACK installation.

Nim being a compiled language means we need to compile the programs mentioned below. It can target a C or C++ backend (among others). The compilation commands differ slightly between the different programs and can depend on usage. The likelihood program below for example can be compiled for the C or C++ backend. In the latter case, the MLP as a classifier is compiled in.

Generally, compilation is done via:

nim c -d:release foo.nim

where foo.nim is any of the programs below. -d:release tells the Nim compiler to compile with optimizations (you can compile with -d:danger for even faster, but less safe, code). Replace c by cpp to compile to the C++ backend.

See the TimepixAnalysis README for further details on how to compile each program.

Unless otherwise specified, each program mentioned below is located in Analysis/ingrid from the root of the TimepixAnalysis repository.

raw_data_manipulation is the first step of the analysis pipeline. Essentially, it is a parsing stage of the data generated by TOS (see section 17.2.1 for an explanation of it) and storing it in a compressed HDF5 (The HDF Group 1997) data file.

The program is fed a directory containing a TOS run via the -p / --path argument. Either a directory containing a single run (i.e. a data taking period ranging typically from minutes to days in length), or a directory that itself contains multiple TOS run directories. Runs compressed as gzipped TAR balls, .tar.gz are also supported.

All data files contained in a run directory will then be parsed in a multithreaded way. The files are memory mapped and parsed in parallel into a Run data structure, which itself contains Event structures.

If FADC files are present in a directory, these will also be parsed into FadcEvent structures in a similar fashion, unless explicitly disabled via the --nofadc option.

Each run is then written into the output HDF5 file as a 'group' (HDF5 terminology). The meta data about each run and event are stored as 'attributes' and additional 'datasets', respectively. The structure of the produced HDF5 file is shown in sec. 34.3.4.1.

In addition, the tool also supports input from HDF5 files containing the raw data from a Timepix3 detector. That data is parsed and reprocessed into the same kind of file structure.

After the raw data has been converted to storage in HDF5, the reconstruction tool is used to start the actual analysis of the data. The program receives an input HDF5 file via the -i / --input argument. As the name implies, the first stage of data analysis is in the form of reconstructing the basic properties of each event. In this stage all events are processed in a multithreaded way. The steps for cluster finding and geometric cluster reconstruction (as mentioned in sec. 9.4) are performed and the data is written to the desired output file given by -o / --outfile.

The produced output HDF5 file then also acts as the input file for reconstruction for all further, optional reconstruction steps. These are mentioned at different parts in the thesis, but we will explain them shortly here now.

--only_fadc

Performs the reconstruction of the FADC data to calculate FADC values such as rise and fall times.

--only_fe_spec

If the input file contains \cefe calibration runs, creates the \cefe spectra and performs fits to them. Also performs the energy calibration for each run.

--only_charge

Performs the ToT calibration of all runs to compute the detected charges in electrons. Requires each chip to be present in the InGrid database (see sec. 34.3.10).

--only_gas_gain

Computes the gas gain in the desired interval lengths via Pólya fits.

--only_gain_fit

If the input file contains \cefe calibration runs, performs the fit of energy calibration runs against the gas gain of each interval. Required to perform energy calibration in background runs.

--only_energy_from_e

Performs the energy calibration for each cluster in the input file.

This is a helper program responsible for the treatment of the X-ray reference data taken at the CAST Detector Lab in Feb. 2019. It receives an HDF5 file as input that is fully reconstructed using raw_data_manipulation and reconstruction containing all runs taken in the CDL. An additional Org table is used as reference to map each run to the correct target/filter kind, found in resources/cdl_runs_2019.org.

The program performs the fits to the correct fluorescence lines for each run based on the target/filter kind in use. It can also produce a helper HDF5 file called calibration-cdl-2018.h5 via the genCdlFile argument, which contains all CDL data split by target/filter kind. This file is used in the context of the likelihood cut method to produce the reference distributions for each cluster property used.

The likelihood program is the (historically named) tool that applies the classifier and any selection of vetoes to an input file. The input files are a fully reconstructed background HDF5 file, corresponding calibration runs and the calibration-cdl-2018.h5 file mentioned above. It has a large number of command line options to adjust the classifier that is used, software efficiency, vetoes, region of the chip to cut to, whether tracking data or background data is selected and more. The program writes the remaining clusters (with additional meta information) to the HDF5 file given by the --h5out argument. The structure is essentially identical to that of the reconstruction tool (the data is stored in a likelihood group instead of a reconstruction group).

The selection of tracking or non-tracking data requires information about when solar trackings took place as attributes inside of the background HDF5 files. These are added using the cast_log_reader, see sec. 34.3.11.

It is also used directly to estimate the random coincidences of the septem and line vetoes, as mentioned in sec. 12.5.5.

The Analysis/ingrid/determineDiffusion directory contains the library / binary to empirically determine the gas diffusion parameters from input data, as explained in sec. 12.4.3. It can either be compiled as a standalone program or be used as a library.

The Analysis/ingrid/nn subdirectory in the TimepixAnalysis repository contains the programs related to the training and evaluation of the multilayer perceptrons (MLP) used in the thesis. Of note is the train_ingrid program, which is the program to train a network. It allows to customize the network to be trained based on command line arguments describing the number of neurons, hidden layers, optimizers, activation functions and so forth. The extended thesis contains the command to train the best performing network.

Secondly, the simulate_xrays program is a helper program to produce an HDF5 file containing simulated X-rays as described in sec. 12.4.2. It makes use of the fake_event_generator.nim file in the ingrid directory, which contains the actual logic.

The InGrid database is both a library part of the TimepixAnalysis repository (InGridDatabase directory) as well as a binary tool and the name for a very simple 'database' storing information about different GridPix chips.

At its core the 'database' part is an HDF5 file containing chip calibrations (ToT, SCurve, …) mapped to timestamps or run numbers in which these are applicable. This allows (mainly) the reconstruction program to retrieve the required calibrations automatically without user input based on the given input files.

To utilize it, the databaseTool needs to be compiled as a binary. Chips are added to the database using this tool. A directory describing the applicable run period and containing calibration files for the chip need to follow the format seen for example in:

https://github.com/Vindaar/TimepixAnalysis/tree/master/resources/ChipCalibrations/Run2

for the Run-2 period of the Septemboard. The runPeriod.toml file describes the applicability of the data, see listing 20 for the file in this case. For each chip, there is simply a directory with the calibration files as produced by TOS and an additional chipInfo.txt file, see listing 21. Note that the runPeriod name needs to match the name of one of the run periods listed in the TOML file.

The databaseTool also allows to perform fits to the calibration data, if needed (for example to analyze SCurves or the raw ToT calibration data).

title = "Run period 2 of CAST, 2017/18"
# list of the run periods defined in the file
runPeriods = ["Run2"]

[Run2]
start = 2017-10-30
stop = 2018-04-11
# either as a sequence of run numbers
validRuns = [ 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89
  90, 91, 92, 93, 94, 95, 96, 97, 98, 99,100,101,102,103
  104,105,106,107,108,109,110,111,112,113,114,115,116,117
  118,119,120,121,122,123,124,125,126,127,128,145,146,147
  148,149,150,151,152,153,154,155,156,157,158,159,160,161
  162,163,164,165,166,167,168,169,170,171,172,173,174,175
  176,177,178,179,180,181,182,183,184,185,186,187,188,189
]
# or as simply a range given as start and stop values
firstRun = 76
lastRun = 189

chipName: H10 W69
runPeriod: Run2
board: SeptemH
chipNumber: 3
Info: This calibration data is valid for Run 2 starting until March 2018!

The LogReader/cast_log_reader is a utility to work with the slow control and tracking log files produced by CAST. It can parse and analyze the different log file formats used over the years and provide different information (magnet operation statistics for example). For the Septemboard detector it provides the option to parse the tracking log files and add the tracking information to the background HDF5 files.

The program first parses the log files and determines valid solar trackings. Then, given an HDF5 data file containing the background data each solar tracking is mapped to a background run. One background run may have zero or more solar trackings attached to it. In the final step the solar tracking start and stop information is added to each run as additional meta data. During the later stages of processing in other TimepixAnalysis programs, notably likelihood, this meta data is then used to only consider background (non tracking) or solar tracking data.

The CAST log files relevant for the Septemboard detector can be found together with the Septemboard CAST data.

mcmc_limit_calculation is the 'final' tool relevant for this thesis. As the name implies it performs the limit calculation using Markov Chain Monte Carlo (MCMC) as explained in detail in chapter 13. See the extended thesis on how to use it.

Multitude of tools for various things that were analyzed over the years. Includes things like computing the gas properties of the Septemboard gas mixture and detection efficiency.

A large number of resources required or simply useful about different data takings, efficiencies, log files and more, which are small enough to be part of a non LFS git repository.

From a practical analysis point of view, the Plotting directory is one of the most interesting parts of the repository. It contains different tools to visualize data at various stages of the analysis pipeline. The most relevant are mentioned briefly here.