All the central parts of the simulation are written in Cython for faster execution. When investigating bugs in this code it is useful to have a debugger. Luckily Cython comes with a built-in debugger, cygdb, but setting up the environment for it to work is non-trivial. The Cython documentation itself does not explain much and is old. The following is a compilation of what I managed to figure out with the help of Mischa Salle.
There needs to be
- A python installation with all the packages necessary to run the code to debug, especially finesse (for finesse debugging) and cython
- A python installation compiled with debug flags of the same version as the first one
The first two need to exist in the same conda environment. That is because the cython code running the debugger needs to find the gdb installation. The python installation with debug symbols only needs to be available somewhere, its path will be manually specified.
How to set up the environment
Assuming an environment set up for finesse development, that is one created from the environment.yml file, cython should already be installed. Otherwise that needs to be done with (after activating the environment)
conda install "cython<3.0"
(Note the version requirement: cython3.0 is out but finesse is currently not compatible with it)
Finesse should also be installed for development work, that means with
Then gdb needs to be installed. This can also be done in conda:
conda install gdb
How to get python with debug flags
To understand the python code the debugger needs a python interpreter that was compiled with debug flags. There are several ways to get this.
Depending on the operating system used there might be a python-dbg/python3-dbg/python3-debug/… available in the package manager. Instead of installing this directly it can also be possible to extract the executable and library from a package (like this debian one) as long as the dependancies are fulfilled on the system and save them somewhere. This python interpreter does not have to be installed as long as it can be run as an executable.
The problem with this approach is that there might not be an interpreter with debug flags matching the python version in the conda environment. A solution to this could be to restrict the python used by conda to versions where a debug build is available.
The alternative is to download the python source code of the version required and make a debug built. See this for how to do that.
How to run the debugger
In the conda environment set up like specified in the finesse directory run
(Note Issue #574. Until this is resolved setup.py line 394 needs to be manually changed to True before running the make)
Then the debugger is ready to run with
cygdb path/to/finesse/directory -- --args path/to/python/debug path/to/file
where path/to/finesse/directory points to the same directory where make was run, path/to/python/debug is the executable of the python debug built and path/to/file is the python script that the debugger is supposed to run.
This will start the gdb debugger. From here on it can be used as explained in the cython documentation.
Building from the research done above (which did not really result in a functioning debugger in the end), I have done some more digging around. The goal was to create the most simple, working cython debugging environment, and subsequently try to get Finesse working inside of that.
To get something that is reproducible on other computers, I wanted the environment running as a docker image. For linux systems I find it easiest to install docker from their apt repository. Quick recap of Docker:
Dockerfiledefines a docker image as set of commands based on other images
- To create an image, you run
Docker build . -t <tag_name>in a directory with a file named
- List images with
- You can run an image as a docker container (a sort of simpler version of a virtual machine) by
docker run --rm -v <local_folder>:<mount_target>/ -it <tag_name> bash. This will drop you in an interactive shell, and the
--rmensures the container will be destroyed when you exit it. The
-vargument lets you mount a folder into the container
After trying for a while to build my own image, following the cython debugging documentation, I found that someone had already created one here (note that you might need to create an account to be able to access this link, but not to actually pull the image). The image is created by running the following script on a base Ubuntu 22.04 image:
apt-get update && apt-get upgrade -y apt-get install -y build-essential git libffi-dev git clone -b 3.10 --depth 1 https://github.com/python/cpython.git /clones/cpython apt-get install -y libssl-dev zlib1g-dev cd /clones/cpython && ./configure --with-pydebug CFLAGS="-g" && make -s -j$(nproc) && make install apt-get install -y wget cd /tmp && wget http://mirrors.kernel.org/sourceware/gdb/releases/gdb-12.1.tar.gz && tar -zxf gdb-12.1.tar.gz apt-get install -y libgmp-dev cd /tmp/gdb-12.1 && ./configure --with-python=python3 && make all install
This clones and compiles python3.10 with debug flags enabled, downloads the
gdb source code and compiles while linking it to the debug python. This differs from the official Cython docs, since they claim it is necessary to have a python2 installation. I have include the Dockerfile I was working on:
FROM ubuntu:22.04 # prevent being asked for a timezone ENV DEBIAN_FRONTEND noninteractive WORKDIR /home # version with the deb-src uncommented, required for apt-get build-dep and apt-get source commands COPY sources.list /etc/apt/ # copy over example cython file COPY helloworld.pyx setup.py test.py /home/ RUN apt-get update RUN apt-get install -y python3-dbg python2-dev curl RUN apt-get -y build-dep gdb RUN apt-get -y source gdb WORKDIR /home/gdb-12.1 RUN ./configure --with-python=python2.7 RUN make RUN make install WORKDIR /home RUN curl https://bootstrap.pypa.io/get-pip.py > get-pip.py RUN curl https://bootstrap.pypa.io/pip/2.7/get-pip.py > get-pip2.py RUN python3-dbg get-pip.py RUN python3-dbg -m pip install cython==0.29.36 RUN python2.7 get-pip2.py RUN python2.7 -m pip install cython==0.29.36 RUN python3-dbg setup.py build_ext --inplace
The main difference is that I installed two python versions following the cython docs and use prebuilt versions from the apt-repository.
Following the instructions of the docker image, it is easy to get the debugging to work as expected. The only downside is that the following statement:
The image does not include Cython – this allows you to install whichever version you would like to use, but of course then requires you do this in your own Dockerfile or in the container while running
Appears to be false. The
0.29.36 version of cython (which is finesse compatible) runs the debugger without complaining, but always only drops you in the C-code, never in the cython code. When you do get into the cython code, trying to read out any variable results into:
Python Exception <type 'exceptions.AttributeError'>: 'PyDictObjectPtr' object has no attribute 'items'
I had the exact same behavior in my own docker image. When you install the latest version of cython (3.0.2), everything works as described in the instructions.