All the central parts of the simulation are written in Cython for faster execution. When investigating bugs in this code it is useful to have a debugger. Luckily Cython comes with a built-in debugger, cygdb, but setting up the environment for it to work is non-trivial. The Cython documentation itself does not explain much and is old. The following is a compilation of what I managed to figure out with the help of Mischa Salle.

Requirements

There needs to be

• A python installation with all the packages necessary to run the code to debug, especially finesse (for finesse debugging) and cython
• gdb
• A python installation compiled with debug flags of the same version as the first one

The first two need to exist in the same conda environment. That is because the cython code running the debugger needs to find the gdb installation. The python installation with debug symbols only needs to be available somewhere, its path will be manually specified.

How to set up the environment

Assuming an environment set up for finesse development, that is one created from the environment.yml file, cython should already be installed. Otherwise that needs to be done with (after activating the environment)

conda install "cython<3.0"

(Note the version requirement: cython3.0 is out but finesse is currently not compatible with it)

Finesse should also be installed for development work, that means with

make develop-conda

Then gdb needs to be installed. This can also be done in conda:

conda install gdb

How to get python with debug flags

To understand the python code the debugger needs a python interpreter that was compiled with debug flags. There are several ways to get this.

Depending on the operating system used there might be a python-dbg/python3-dbg/python3-debug/… available in the package manager. Instead of installing this directly it can also be possible to extract the executable and library from a package (like this debian one) as long as the dependancies are fulfilled on the system and save them somewhere. This python interpreter does not have to be installed as long as it can be run as an executable.

The problem with this approach is that there might not be an interpreter with debug flags matching the python version in the conda environment. A solution to this could be to restrict the python used by conda to versions where a debug build is available.

The alternative is to download the python source code of the version required and make a debug built. See this for how to do that.

How to run the debugger

In the conda environment set up like specified in the finesse directory run

make debug

(Note Issue #574. Until this is resolved setup.py line 394 needs to be manually changed to True before running the make)

Then the debugger is ready to run with

cygdb path/to/finesse/directory -- --args path/to/python/debug path/to/file

where path/to/finesse/directory points to the same directory where make was run, path/to/python/debug is the executable of the python debug built and path/to/file is the python script that the debugger is supposed to run.

This will start the gdb debugger. From here on it can be used as explained in the cython documentation.

Update [Miron]

Building from the research done above (which did not really result in a functioning debugger in the end), I have done some more digging around. The goal was to create the most simple, working cython debugging environment, and subsequently try to get Finesse working inside of that.

To get something that is reproducible on other computers, I wanted the environment running as a docker image. For linux systems I find it easiest to install docker from their apt repository. Quick recap of Docker:

• A Dockerfile defines a docker image as set of commands based on other images
• To create an image, you run Docker build . -t <tag_name> in a directory with a file named Dockerfile
• List images with docker images
• You can run an image as a docker container (a sort of simpler version of a virtual machine) by docker run --rm -v <local_folder>:<mount_target>/ -it <tag_name> bash. This will drop you in an interactive shell, and the --rm ensures the container will be destroyed when you exit it. The -v argument lets you mount a folder into the container

After trying for a while to build my own image, following the cython debugging documentation, I found that someone had already created one here (note that you might need to create an account to be able to access this link, but not to actually pull the image). The image is created by running the following script on a base Ubuntu 22.04 image:

apt-get update && apt-get upgrade -y
apt-get install -y build-essential git libffi-dev
git clone -b 3.10 --depth 1 https://github.com/python/cpython.git /clones/cpython
apt-get install -y libssl-dev zlib1g-dev
cd /clones/cpython && ./configure --with-pydebug CFLAGS="-g" && make -s -j$(nproc) && make install
apt-get install -y wget
cd /tmp && wget http://mirrors.kernel.org/sourceware/gdb/releases/gdb-12.1.tar.gz && tar -zxf gdb-12.1.tar.gz
apt-get install -y libgmp-dev
cd /tmp/gdb-12.1 && ./configure --with-python=python3 && make all install

This clones and compiles python3.10 with debug flags enabled, downloads the gdb source code and compiles while linking it to the debug python. This differs from the official Cython docs, since they claim it is necessary to have a python2 installation. I have include the Dockerfile I was working on:

FROM ubuntu:22.04

# prevent being asked for a timezone
ENV DEBIAN_FRONTEND noninteractive

WORKDIR /home
# version with the deb-src uncommented, required for apt-get build-dep and apt-get source commands
COPY sources.list /etc/apt/
# copy over example cython file
COPY helloworld.pyx setup.py test.py /home/

RUN apt-get update
RUN apt-get install -y python3-dbg python2-dev curl
RUN apt-get -y build-dep gdb
RUN apt-get -y source gdb

WORKDIR /home/gdb-12.1
RUN ./configure --with-python=python2.7
RUN make
RUN make install

WORKDIR /home
RUN curl https://bootstrap.pypa.io/get-pip.py > get-pip.py
RUN curl https://bootstrap.pypa.io/pip/2.7/get-pip.py > get-pip2.py
RUN python3-dbg get-pip.py
RUN python3-dbg -m pip install cython==0.29.36
RUN python2.7 get-pip2.py
RUN python2.7 -m pip install cython==0.29.36
RUN python3-dbg setup.py build_ext --inplace

The main difference is that I installed two python versions following the cython docs and use prebuilt versions from the apt-repository.

Following the instructions of the docker image, it is easy to get the debugging to work as expected. The only downside is that the following statement:

The image does not include Cython – this allows you to install whichever version you would like to use, but of course then requires you do this in your own Dockerfile or in the container while running

Appears to be false. The 0.29.36 version of cython (which is finesse compatible) runs the debugger without complaining, but always only drops you in the C-code, never in the cython code. When you do get into the cython code, trying to read out any variable results into:

Python Exception <type 'exceptions.AttributeError'>: 'PyDictObjectPtr' object has no attribute 'items'

I had the exact same behavior in my own docker image. When you install the latest version of cython (3.0.2), everything works as described in the instructions.

Some more useful links:
https://willayd.com/fundamental-python-debugging-part-3-cython-extensions.html
https://github.com/cython/cython/pull/5186