Re: [ptxdist] [PATCH v2] doc: daily work: update debugging section

[Michael Olbrich <m.olbrich@pengutronix.de>]

On Mon, Aug 26, 2024 at 02:04:18PM +0200, Roland Hieber wrote:
> * Split off the "Running QEMU" section into a stand-alone section
> * Mention /usr/lib/debug instead of the old ./debug/ folders
> * Describe `ptxdist gdb` and debugging on the target directly
> * Keep GDB usage short; refer to third-party documentation
> * Use literal blocks instead of code blocks without highlighting
> * Make use of the |ptxdistPlatformDir| template variable
> 
> Signed-off-by: Roland Hieber <rhi@pengutronix.de>
> ---
> PATCH v2:
> * somehow I forgot to stage & amend various changes before sending v1,
>   please disregard the old one
> 
> 
>  doc/daily_work.inc | 226 ++++++++++++++++++++++++++++++++++-----------
>  1 file changed, 174 insertions(+), 52 deletions(-)
> 
> diff --git a/doc/daily_work.inc b/doc/daily_work.inc
> index b98adf75197d..65b0bad25d97 100644
> --- a/doc/daily_work.inc
> +++ b/doc/daily_work.inc
> @@ -285,15 +285,11 @@ nonexisting file, we can limit the output to all ``open`` system calls:
>  The binary may fail due to a missing ``/etc/foo.conf``. This could be a
>  hint on what is going wrong (it might not be the final solution).
>  
> -Debugging with CPU emulation
> -----------------------------
> -
> -If we do not need some target related feature to run our application, we
> -can also debug it through a simple CPU emulation. Thanks to QEMU we can
> -run ELF binaries for other architectures than our build host is.
> +Running an Application Made for a Different Architecture
> +--------------------------------------------------------
>  
> -Running an Application made for a different Architecture
> -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +Thanks to QEMU we can run ELF binaries for different architectures than our
> +build host.
>  
>  PTXdist creates a fully working root filesystem with all run-time
>  components in ``root/``. Lets assume we made a PTXdist based project for
> @@ -318,64 +314,134 @@ host’s operating system. Using QEMU in this way let us simply check our
>  programs. There are also QEMU environments for other architectures
>  available.
>  
> -Debugging an Application made for a different Architecture
> -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> -
> -Debugging our application is also possible with QEMU. All we need are a
> -root filesystem with debug symbols available, QEMU and an architecture
> -aware debugger.
> -
> -The root filesystem with debug symbols will be provided by PTXdist, the
> -architecture aware debugger comes with the OSELAS.Toolchain. Two
> -consoles are required for this debug session in this example. We start
> -the QEMU in the first console as:
> -
> -.. code-block:: text
> -
> -    $ cd ptxdistPlatformDir/root
> -    ptxdistPlatformDir/root$ qemu-<architecture> -g 1234 -cpu <cpu-core> -L . usr/bin/myapp
> +Debugging an Application Made for a Different Architecture
> +----------------------------------------------------------
> +
> +Debugging an application is possible in three ways:
> +
> +#. :ref:`Directly running GDB on the target hardware <gdb_on_target>`:
> +   This approach has the advantage that the target system is "self-contained",
> +   but it is necessary to install the debug symbols in the target's root file
> +   system, which can significantly improve the memory footprint. Furthermore,

"improve" sounds wrong here. Maybe you meant "increase"?

> +   it is often not possible for GDB to access the application's source code
> +   unless further steps are taken (e.g. making the source code available in the
> +   correct paths over an NFS share or from a USB drive).
> +
> +#. :ref:`Running the application in a QEMU emulating the target architecture <gdb_via_qemu>`:
> +   This approach doesn't require any additional hardware and can therefore be
> +   done completely on the build host, but it is limited to applications
> +   that do not depend on specific behaviour of the target hardware, like
> +   using external interfaces or architecture-specific features that are not
> +   available via QEMU (e.g. accessing GPS hardware over UART, or graphics
> +   acceleration).
> +
> +#. :ref:`Running gdbserver on the target hardware, and connecting to it with GDB
> +   from the build host <ptxdist_gdb>`:
> +   This is the simplest use case when a target hardware with network interfaces
> +   is available, and PTXdist includes native support for it.
> +   In this case, the target file system only has to include the *gdbserver*
> +   binary, but does not need to contain any further debug information or source
> +   code, which are supplied by the build host.
> +
> +In any case, we need the respective binaries including their debug information,
> +and a debugger for the target architecture.
> +The root filesystem with debug symbols is provided by PTXdist in
> +``|ptxdistPlatformDir|/root/`` on the build host, and the architecture-aware debugger
> +comes with the OSELAS.Toolchain.
> +
> +.. note:: When PTXdist builds a target root file system, the binaries that are
> +   installed into it during the *targetinstall* stage of each package are
> +   always stripped of their debug symbols.  Instead, the debug symbols are
> +   installed under ``|ptxdistPlatformDir|/root/usr/lib/debug/`` in separate files named
> +   after the *build ID* of the binary (which is a checksum over the contents of
> +   the binary, and can change when a binary is rebuilt).
> +
> +   When GDB loads a binary, it will look for and automatically load matching
> +   debug symbols in ``usr/lib/debug`` relative to its *sysroot*, which is
> +   usually ``/``, but can be configured at runtime.
> +
> +.. note:: Information on using GDB is not part of this manual.
> +   There are countless cheatsheets available on the internet, but in most
> +   cases you will get around sufficiently with the ``break <location|function>``,
> +   ``run``, ``next``, ``step``, ``finish``, ``continue``, ``print <statement|address>``,
> +   ``list <location|function>`` and ``quit`` commands.
> +   Current GDB versions also include a well-structured online help with the
> +   ``help`` command, which can show you how to use each command.
> +
> +.. _gdb_on_target:
> +
> +Running GDB on the Target Hardware
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +In this case, GDB and the debug information need to be available on the target hardware.
> +GDB can be enabled in the *menuconfig* with the ``PTXCONF_GDB`` symbol.
> +Additionally, the debug symbols are installed into the target root file system 
> +when the ``PTXCONF_DEBUG_PACKAGES_INSTALL`` option is enabled.
> +(If an image has been built before, i.e. the *targetinstall* stages have
> +already been executed, a ``ptxdist clean root`` followed by a ``ptxdist go`` or
> +``ptxdist images`` might be needed to install all debug symbols.)
> +
> +Due to the size of the resulting root file system, the target root file system
> +is often supplied via :ref:`NFSROOT <nfsroot>`.
> +
> +On the target, GDB can then be used normally (with the caveat that the source files are
> +not available unless supplied manually)::
> +
> +   root@target $ gdb /usr/bin/systemctl 
> +   Reading symbols from /usr/bin/systemctl...
> +   Reading symbols from /usr/lib/debug/.build-id/fb/3b770e637b51276dcdc41a1e8b37df8a6d6686.debug...
> +   (gdb) break main
> +   Breakpoint 1 at 0x6c90: file ../systemd-256.1/src/systemctl/systemctl.c, line 1372.
> +   (gdb) run
> +   Starting program: /usr/bin/systemctl 
> +   [Thread debugging using libthread_db enabled]
> +   Using host libthread_db library "/lib/libthread_db.so.1".
> +
> +   Breakpoint 1, main (argc=1, argv=0xbee9bd64)
> +       at ../systemd-256.1/src/systemctl/systemctl.c:1372
> +   1372    in ../systemd-256.1/src/systemctl/systemctl.c
> +   (gdb) backtrace
> +   #0  main (argc=1, argv=0xbee9bd64)
> +       at ../systemd-256.1/src/systemctl/systemctl.c:1372
> +   (gdb) list
> +   1372    ../systemd-256.1/src/systemctl/systemctl.c: No such file or directory.
> +   (gdb) quit
> +
> +.. _gdb_via_qemu:
> +
> +Debugging with CPU Emulation
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +If we do not need target-specific features to run our application, we can also
> +debug it through CPU emulation.
> +Two consoles are required for this debug session in this example. We start
> +the QEMU in the first console as::
>  
> -.. note:: PTXdist always builds a root filesystem ``root/``.
> -  It contains all components without debug 
> -  information (all binaries are in the same size as used later on on the
> -  real target). In addition, each directory that contains binaries also
> -  contains a ``.debug/`` directory. It contains a file with only the debug
> -  symbols for each binary. These files are ignored while running
> -  applications but GDB knows about it and will automatically load the debug
> -  files.
> +    $ cd |ptxdistPlatformDir|/root
> +    |ptxdistPlatformDir|/root$ qemu-<architecture> -g 1234 -cpu <cpu-core> -L . usr/bin/myapp
>  
>  The added *-g 1234* parameter lets QEMU wait for a GDB connection to run
>  the application.
>  
>  In the second console we start GDB with the correct architecture
>  support. This GDB comes with the same OSELAS.Toolchain that was also
> -used to build the project:
> -
> -.. code-block:: text
> +used to build the project::
>  
> -    $ ./selected_toolchain/<target>-gdb --tui platform-<platformname>/root/usr/bin/myapp
> -
> -This will run a *curses* based GDB. Not so easy to handle (we must enter
> -all the commands and cannot click with a mouse!), but very fast to take
> -a quick look at our application.
> +    $ ./selected_toolchain/<target>-gdb |ptxdistPlatformDir|/root/usr/bin/myapp
>  
>  At first we tell GDB where to look for debug symbols. The correct
> -directory here is ``root/``.
> -
> -.. code-block:: text
> +directory here is ``|ptxdistPlatformDir|/root/``::
>  
> -    (gdb) set solib-absolute-prefix platform-<platformname>/root
> +    (gdb) set solib-absolute-prefix |ptxdistPlatformDir|/root

This part of the documentation is pretty old and a bit outdated. Can you
change it to use 'ptxdist gdb'? It should possible to use it in this case
and then manually setting 'solib-absolute-prefix' is not needed.

Michael

>  
> -Next we connect this GDB to the waiting QEMU:
> -
> -.. code-block:: text
> +Next we connect this GDB to the waiting QEMU::
>  
>      (gdb) target remote localhost:1234
>      Remote debugging using localhost:1234
>      [New Thread 1]
>      0x40096a7c in _start () from root/lib/ld.so.1
>  
> -As our application is already started, we can’t use the GDB command
> +As our application is already started via QEMU, we can’t use the GDB command
>  ``start`` to run it until it reaches ``main()``. We set a breakpoint
>  instead at ``main()`` and *continue* the application:
>  
> @@ -387,10 +453,6 @@ instead at ``main()`` and *continue* the application:
>      Continuing.
>      Breakpoint 1, main (argc=1, argv=0x4007f03c) at myapp.c:644
>  
> -The top part of the running gdbtui console will always show us the
> -current source line. Due to the ``root/`` directory usage all
> -debug information for GDB is available.
> -
>  Now we can step through our application by using the commands *step*,
>  *next*, *stepi*, *nexti*, *until* and so on.
>  
> @@ -403,6 +465,66 @@ Now we can step through our application by using the commands *step*,
>    debug symbols kept, it will be also possible for GDB to debug C library
>    functions our application calls (so it might worth the disk space).
>  
> +.. _ptxdist_gdb:
> +
> +Remote-Debugging with *gdbserver*
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +When debugging with *gdbserver*, the debugging session is split into two parts:
> +the server runs on the target and is remote-controlled by a GDB instance
> +running on the build host over a network connection.
> +You need to enable the ``PTXCONF_GDBSERVER`` symbol in PTXdists's *menuconfig*
> +which makes *gdbserver* available on your target.
> +
> +Like for the approach above, you will need two terminals.
> +In the first terminal you connect to your running target (e.g. via SSH or
> +serial console), start *gdbserver* and tell it the binary to debug and where
> +to listen for connections::
> +
> +   root@target $ gdbserver 192.168.1.11:1234 /usr/bin/systemctl
> +   Process /usr/bin/systemctl created; pid = 477
> +   Listening on port 1234
> +
> +In the above example, 192.168.1.11 is the outside-facing IP address of the
> +target, and 1234 is the port on which to listen.
> +gdbserver has now created a process, and waits for connections from a GDB.
> +
> +In the other terminal, you then start ``ptxdist gdb``, which will set up GDB
> +correctly so that it can find all necessary debug symbols in PTXdist's
> +``|ptxdistPlatformDir|/root/`` folder, and the sources in
> +``|ptxdistPlatformDir|/sysroot-target/``.
> +You then connect to the gdbserver instance running on the target using GDB's
> +*target remote* command::
> +
> +   user@buildhost $ ptxdist gdb
> +   (gdb) target remote 192.168.9.10:1234
> +   Remote debugging using 192.168.9.10:1234
> +   Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/bin/systemctl...
> +   Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/lib/debug/.build-id/fb/3b770e637b51276dcdc41a1e8b37df8a6d6686.debug...
> +   Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/lib/ld-linux-armhf.so.3...
> +   Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/lib/debug/.build-id/0d/a8d3736dcd31fbf43ea0cb79d1f9699c6695ff.debug...
> +   0xb6f29624 in _start () from …/my-bsp/|ptxdistPlatformDir|/root/lib/ld-linux-armhf.so.3
> +   (gdb) break main
> +   Breakpoint 1 at 0x4e3c90: file ../systemd-256.1/src/systemctl/systemctl.c, line 1372.
> +   (gdb) continue
> +   Continuing.
> +
> +   Breakpoint 1, main (argc=1, argv=0xbe829d84) at ../systemd-256.1/src/systemctl/systemctl.c:1372
> +   1372    DEFINE_MAIN_FUNCTION_WITH_POSITIVE_FAILURE(run);
> +   (gdb) list
> +   1367    
> +   1368            /* Note that we return r here, not 0, so that we can implement the LSB-like return codes */
> +   1369            return r;
> +   1370    }
> +   1371    
> +   1372    DEFINE_MAIN_FUNCTION_WITH_POSITIVE_FAILURE(run);
> +   1373    #endif
> +   (gdb)
> +
> +You can now use GDB as usual, and when the debugged process terminates (or you
> +quit the GDB session), *gdbserver* will terminate as well, and close the
> +connection.
> +
>  Migration between Releases
>  --------------------------
>  
> -- 
> 2.39.2
> 
> 
> 

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |