{"id":335,"date":"2023-08-31T12:44:01","date_gmt":"2023-08-31T10:44:01","guid":{"rendered":"https:\/\/logbooks.ifosim.org\/finesse\/?p=335"},"modified":"2023-09-11T17:04:11","modified_gmt":"2023-09-11T15:04:11","slug":"how-to-set-up-the-cython-debugger","status":"publish","type":"post","link":"https:\/\/logbooks.ifosim.org\/finesse\/2023\/08\/31\/how-to-set-up-the-cython-debugger\/","title":{"rendered":"How to set up the Cython debugger"},"content":{"rendered":"\n<p>All the central parts of the simulation are written in Cython for faster execution. When investigating bugs in this code it is useful to have a debugger. Luckily Cython comes with a built-in debugger, cygdb, but setting up the environment for it to work is non-trivial. The <a href=\"https:\/\/cython.readthedocs.io\/en\/latest\/src\/userguide\/debugging.html\" data-type=\"link\" data-id=\"https:\/\/cython.readthedocs.io\/en\/latest\/src\/userguide\/debugging.html\">Cython documentation<\/a> itself does not explain much and is old. The following is a compilation of what I managed to figure out with the help of Mischa Salle.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Requirements<\/h2>\n\n\n\n<p>There needs to be<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A python installation with all the packages necessary to run the code to debug, especially finesse (for finesse debugging) and cython<\/li>\n\n\n\n<li>gdb<\/li>\n\n\n\n<li>A python installation compiled with debug flags of the same version as the first one<\/li>\n<\/ul>\n\n\n\n<p>The first two need to exist in the same conda environment. That is because the cython code running the debugger needs to find the gdb installation. The python installation with debug symbols only needs to be available somewhere, its path will be manually specified.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How to set up the environment<\/h2>\n\n\n\n<p>Assuming an environment set up for finesse development, that is one created from the environment.yml file, cython should already be installed. Otherwise that needs to be done with (after activating the environment)<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>conda install \"cython&lt;3.0\"<\/code><\/pre>\n\n\n\n<p>(Note the version requirement: cython3.0 is out but finesse is currently not compatible with it)<\/p>\n\n\n\n<p>Finesse should also be installed for development work, that means with<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>make develop-conda<\/code><\/pre>\n\n\n\n<p>Then gdb needs to be installed. This can also be done in conda:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>conda install gdb<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">How to get python with debug flags<\/h2>\n\n\n\n<p>To understand the python code the debugger needs a python interpreter that was compiled with debug flags. There are several ways to get this.<\/p>\n\n\n\n<p>Depending on the operating system used there might be a python-dbg\/python3-dbg\/python3-debug\/&#8230; available in the package manager. Instead of installing this directly it can also be possible to extract the executable and library from a package (like <a href=\"https:\/\/packages.debian.org\/bookworm\/debug\/python3-dbg\" data-type=\"link\" data-id=\"https:\/\/packages.debian.org\/bookworm\/debug\/python3-dbg\">this debian one<\/a>) as long as the dependancies are fulfilled on the system and save them somewhere. This python interpreter does not have to be installed as long as it can be run as an executable.<\/p>\n\n\n\n<p>The problem with this approach is that there might not be an interpreter with debug flags matching the python version in the conda environment. A solution to this could be to restrict the python used by conda to versions where a debug build is available.<\/p>\n\n\n\n<p>The alternative is to download the python source code of the version required and make a debug built. See <a href=\"https:\/\/devguide.python.org\/getting-started\/setup-building\/\" data-type=\"link\" data-id=\"https:\/\/devguide.python.org\/getting-started\/setup-building\/\">this<\/a> for how to do that.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How to run the debugger<\/h2>\n\n\n\n<p>In the conda environment set up like specified in the finesse directory run<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>make debug<\/code><\/pre>\n\n\n\n<p>(Note <a href=\"https:\/\/gitlab.com\/ifosim\/finesse\/finesse3\/-\/issues\/574\" data-type=\"link\" data-id=\"https:\/\/gitlab.com\/ifosim\/finesse\/finesse3\/-\/issues\/574\">Issue #574<\/a>. Until this is resolved setup.py line 394 needs to be manually changed to True before running the make)<\/p>\n\n\n\n<p>Then the debugger is ready to run with<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cygdb path\/to\/finesse\/directory -- --args path\/to\/python\/debug path\/to\/file<\/code><\/pre>\n\n\n\n<p>where path\/to\/finesse\/directory points to the same directory where make was run, path\/to\/python\/debug is the executable of the python debug built and path\/to\/file is the python script that the debugger is supposed to run.<\/p>\n\n\n\n<p>This will start the gdb debugger. From here on it can be used as explained in the <a href=\"https:\/\/cython.readthedocs.io\/en\/latest\/src\/userguide\/debugging.html#using-the-debugger\" data-type=\"link\" data-id=\"https:\/\/cython.readthedocs.io\/en\/latest\/src\/userguide\/debugging.html#using-the-debugger\">cython documentation<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Update [Miron]<\/h2>\n\n\n\n<p>Building from the research done above (which did not really result in a functioning debugger in the end), I have done some more digging around. The goal was to create the most simple, working cython debugging environment, and subsequently try to get Finesse working inside of that.<\/p>\n\n\n\n<p>To get something that is reproducible on other computers, I wanted the environment running as a <a href=\"https:\/\/docs.docker.com\/\">docker<\/a> image. For linux systems I find it easiest to <a href=\"https:\/\/docs.docker.com\/engine\/install\/ubuntu\/#install-using-the-repository\">install docker from their apt repository<\/a>. Quick recap of Docker:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A <code>Dockerfile<\/code> defines a docker image as set of commands based on other images<\/li>\n\n\n\n<li>To create an image, you run <code>Docker build . -t &lt;tag_name&gt;<\/code> in a directory with a file named <code>Dockerfile<\/code><\/li>\n\n\n\n<li>List images with <code>docker images<\/code><\/li>\n\n\n\n<li>You can run an image as a docker container (a sort of simpler version of a virtual machine) by <code>docker run --rm -v &lt;local_folder&gt;:&lt;mount_target&gt;\/ -it &lt;tag_name&gt; bash<\/code>. This will drop you in an interactive shell, and the <code>--rm<\/code> ensures the container will be destroyed when you exit it. The <code>-v<\/code> argument lets you mount a folder into the container<\/li>\n<\/ul>\n\n\n\n<p>After trying for a while to build my own image, following the <a href=\"https:\/\/cython.readthedocs.io\/en\/latest\/src\/userguide\/debugging.html\">cython debugging documentation,<\/a> I found that someone had already created one <a href=\"https:\/\/hub.docker.com\/r\/willayd\/cython-debug\">here<\/a> (note that you might need to create an account to be able to access this link, but not to actually pull the image). The image is created by running the following script on a base Ubuntu 22.04 image:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>apt-get update &amp;&amp; apt-get upgrade -y\napt-get install -y build-essential git libffi-dev\ngit clone -b 3.10 --depth 1 https:\/\/github.com\/python\/cpython.git \/clones\/cpython\napt-get install -y libssl-dev zlib1g-dev\ncd \/clones\/cpython &amp;&amp; .\/configure --with-pydebug CFLAGS=\"-g\" &amp;&amp; make -s -j$(nproc) &amp;&amp; make install\napt-get install -y wget\ncd \/tmp &amp;&amp; wget http:\/\/mirrors.kernel.org\/sourceware\/gdb\/releases\/gdb-12.1.tar.gz &amp;&amp; tar -zxf gdb-12.1.tar.gz\napt-get install -y libgmp-dev\ncd \/tmp\/gdb-12.1 &amp;&amp; .\/configure --with-python=python3 &amp;&amp; make all install<\/code><\/pre>\n\n\n\n<p>This clones and compiles python3.10 with debug flags enabled, downloads the <code>gdb<\/code> source code and compiles while linking it to the debug python. This differs from the official Cython docs, since they claim it is necessary to have a python2 installation. I have include the Dockerfile I was working on:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>FROM ubuntu:22.04\n\n# prevent being asked for a timezone\nENV DEBIAN_FRONTEND noninteractive\n\nWORKDIR \/home\n# version with the deb-src uncommented, required for apt-get build-dep and apt-get source commands\nCOPY sources.list \/etc\/apt\/\n# copy over example cython file\nCOPY helloworld.pyx setup.py test.py \/home\/\n\nRUN apt-get update\nRUN apt-get install -y python3-dbg python2-dev curl\nRUN apt-get -y build-dep gdb\nRUN apt-get -y source gdb\n\nWORKDIR \/home\/gdb-12.1\nRUN .\/configure --with-python=python2.7\nRUN make\nRUN make install\n\nWORKDIR \/home\nRUN curl https:\/\/bootstrap.pypa.io\/get-pip.py &gt; get-pip.py\nRUN curl https:\/\/bootstrap.pypa.io\/pip\/2.7\/get-pip.py &gt; get-pip2.py\nRUN python3-dbg get-pip.py\nRUN python3-dbg -m pip install cython==0.29.36\nRUN python2.7 get-pip2.py\nRUN python2.7 -m pip install cython==0.29.36\nRUN python3-dbg setup.py build_ext --inplace<\/code><\/pre>\n\n\n\n<p>The main difference is that I installed two python versions following the cython docs and use prebuilt versions from the apt-repository.<\/p>\n\n\n\n<p>Following the <a href=\"https:\/\/hub.docker.com\/r\/willayd\/cython-debug\">instructions<\/a> of the docker image, it is easy to get the debugging to work as expected. The only downside is that the following statement:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p><\/p>\n<cite>The image does not include Cython &#8211; this allows you to install whichever version you would like to use, but of course then requires you do this in your own Dockerfile or in the container while running<\/cite><\/blockquote>\n\n\n\n<p>Appears to be false. The <code>0.29.36<\/code> version of cython (which is finesse compatible) runs the debugger without complaining, but always only drops you in the C-code, never in the cython code. When you do get into the cython code, trying to read out any variable results into:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Python Exception &lt;type 'exceptions.AttributeError'&gt;: 'PyDictObjectPtr' object has no attribute 'items'<\/code><\/pre>\n\n\n\n<p>I had the exact same behavior in my own docker image. When you install the latest version of cython (3.0.2), everything works as described in the instructions.<\/p>\n\n\n\n<p>Some more useful links:<br><a href=\"https:\/\/willayd.com\/fundamental-python-debugging-part-3-cython-extensions.html\">https:\/\/willayd.com\/fundamental-python-debugging-part-3-cython-extensions.html<\/a><br><a href=\"https:\/\/github.com\/cython\/cython\/pull\/5186\">https:\/\/github.com\/cython\/cython\/pull\/5186<\/a><\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>All the central parts of the simulation are written in Cython for faster execution. When investigating bugs in this code it is useful to have a debugger. Luckily Cython comes with a built-in debugger, cygdb, but setting up the environment for it to work is non-trivial. The Cython documentation itself does not explain much and [&hellip;]<\/p>\n","protected":false},"author":73,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ssl_alp_hide_revisions":false,"footnotes":"","ssl_alp_hide_crossreferences_to":false},"categories":[1],"tags":[78],"ssl-alp-coauthor":[42],"class_list":["post-335","post","type-post","status-publish","format-standard","hentry","category-uncategorised","tag-debugging"],"_links":{"self":[{"href":"https:\/\/logbooks.ifosim.org\/finesse\/wp-json\/wp\/v2\/posts\/335","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/logbooks.ifosim.org\/finesse\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/logbooks.ifosim.org\/finesse\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/logbooks.ifosim.org\/finesse\/wp-json\/wp\/v2\/users\/73"}],"replies":[{"embeddable":true,"href":"https:\/\/logbooks.ifosim.org\/finesse\/wp-json\/wp\/v2\/comments?post=335"}],"version-history":[{"count":3,"href":"https:\/\/logbooks.ifosim.org\/finesse\/wp-json\/wp\/v2\/posts\/335\/revisions"}],"predecessor-version":[{"id":339,"href":"https:\/\/logbooks.ifosim.org\/finesse\/wp-json\/wp\/v2\/posts\/335\/revisions\/339"}],"wp:attachment":[{"href":"https:\/\/logbooks.ifosim.org\/finesse\/wp-json\/wp\/v2\/media?parent=335"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/logbooks.ifosim.org\/finesse\/wp-json\/wp\/v2\/categories?post=335"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/logbooks.ifosim.org\/finesse\/wp-json\/wp\/v2\/tags?post=335"},{"taxonomy":"ssl-alp-coauthor","embeddable":true,"href":"https:\/\/logbooks.ifosim.org\/finesse\/wp-json\/wp\/v2\/ssl-alp-coauthor?post=335"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}