[ptxdist] [PATCH] doc: daily work: update debugging section

[ptxdist] [PATCH] doc: daily work: update debugging section

[Roland Hieber]

* Split off the "Running QEMU" section into a stand-alone section
* Mention /usr/lib/debug instead of the old ./debug/ folders
* Describe `ptxdist gdb` and debugging on the target directly
* Keep GDB usage short; refer to third-party documentation
* Use literal blocks instead of code blocks without highlighting
* Make use of the |ptxdistPlatformDir| template variable

Signed-off-by: Roland Hieber <rhi@pengutronix.de>
---
 doc/daily_work.inc | 224 ++++++++++++++++++++++++++++++++++-----------
 1 file changed, 172 insertions(+), 52 deletions(-)

diff --git a/doc/daily_work.inc b/doc/daily_work.inc
index b98adf75197d..d0c0d8bad922 100644
--- a/doc/daily_work.inc
+++ b/doc/daily_work.inc
@@ -285,15 +285,11 @@ nonexisting file, we can limit the output to all ``open`` system calls:
 The binary may fail due to a missing ``/etc/foo.conf``. This could be a
 hint on what is going wrong (it might not be the final solution).
 
-Debugging with CPU emulation
-----------------------------
-
-If we do not need some target related feature to run our application, we
-can also debug it through a simple CPU emulation. Thanks to QEMU we can
-run ELF binaries for other architectures than our build host is.
+Running an Application Made for a Different Architecture
+--------------------------------------------------------
 
-Running an Application made for a different Architecture
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Thanks to QEMU we can run ELF binaries for different architectures than our
+build host.
 
 PTXdist creates a fully working root filesystem with all run-time
 components in ``root/``. Lets assume we made a PTXdist based project for
@@ -318,64 +314,134 @@ host’s operating system. Using QEMU in this way let us simply check our
 programs. There are also QEMU environments for other architectures
 available.
 
-Debugging an Application made for a different Architecture
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Debugging our application is also possible with QEMU. All we need are a
-root filesystem with debug symbols available, QEMU and an architecture
-aware debugger.
-
-The root filesystem with debug symbols will be provided by PTXdist, the
-architecture aware debugger comes with the OSELAS.Toolchain. Two
-consoles are required for this debug session in this example. We start
-the QEMU in the first console as:
-
-.. code-block:: text
-
-    $ cd ptxdistPlatformDir/root
-    ptxdistPlatformDir/root$ qemu-<architecture> -g 1234 -cpu <cpu-core> -L . usr/bin/myapp
-
-.. note:: PTXdist always builds a root filesystem ``root/``.
-  It contains all components without debug 
-  information (all binaries are in the same size as used later on on the
-  real target). In addition, each directory that contains binaries also
-  contains a ``.debug/`` directory. It contains a file with only the debug
-  symbols for each binary. These files are ignored while running
-  applications but GDB knows about it and will automatically load the debug
-  files.
+Debugging an Application Made for a Different Architecture
+----------------------------------------------------------
+
+Debugging an application is possible in three ways:
+
+#. :ref:`Directly running GDB on the target hardware <gdb_on_target>`:
+   This approach has the advantage that the target system is "self-contained",
+   but it is necessary to install the debug symbols in the target's root file
+   system, which can significantly improve the memory footprint. Furthermore,
+   it is often not possible for GDB to access the application's source code
+   unless further steps are taken (e.g. making the source code available in the
+   correct paths over an NFS share or from a USB drive).
+
+#. :ref:`Running the application in a QEMU emulating the target architecture <gdb_via_qemu>`:
+   This approach doesn't require any additional hardware and can therefore be
+   done completely on the build host, but it is limited to applications
+   that do not depend on specific behaviour of the target hardware, like
+   using external interfaces or architecture-specific features that are not
+   available via QEMU (e.g. accessing GPS hardware over UART, or graphics
+   acceleration).
+
+#. :ref:`Running *gdbserver* on the target hardware, and connecting to it with GDB
+   from the build host <ptxdist_gdb>`:
+   This is the simplest use case when a target hardware with network interfaces
+   is available, and PTXdist includes native support for it.
+   In this case, the target file system only has to include the *gdbserver*
+   binary, but does not need to contain any further debug information or source
+   code, which are supplied by the build host.
+
+In any case, we need the respective binaries including their debug information,
+and a debugger for the target architecture.
+The root filesystem with debug symbols is provided by PTXdist in
+``platform-nnn/root/`` on the build host, and the architecture-aware debugger
+comes with the OSELAS.Toolchain.
+
+.. note:: When PTXdist builds a target root file system, the binaries that are
+   installed into it during the *targetinstall* stage of each package are
+   always stripped of their debug symbols.  Instead, the debug symbols are
+   installed under ``platform-nnn/root/usr/lib/debug/`` in separate files named
+   after the *build ID* of the binary (which is a checksum over the contents of
+   the binary, and can change when a binary is rebuilt).
+
+   When GDB loads a binary, it will look for and automatically load matching
+   debug symbols in ``usr/lib/debug`` relative to its *sysroot*, which is
+   usually ``/``, but can be configured at runtime.
+
+.. note:: Information on using GDB is not part of this manual.
+   There are countless cheatsheets available on the internet, but in most
+   cases you will get around sufficiently with the ``break <location|function>``,
+   ``run``, ``next``, ``step``, ``finish``, ``continue``, ``print <statement|address>``,
+   ``list <location|function>`` and ``quit`` commands.
+   Current GDB versions also include a well-structured online help with the
+   ``help`` command, which can show you how to use each command.
+
+.. _gdb_on_target:
+
+Running GDB on the Target Hardware
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In this case, GDB and the debug information need to be available on the target hardware.
+GDB can be enabled in the *menuconfig* with the *PTXCONF_GDB* symbol.
+Additionally, the debug symbols are installed into the target root file system 
+when the ``PTXCONF_DEBUG_PACKAGES_INSTALL`` option is enabled.
+(If an image has been built before, i.e. the *targetinstall* stages have
+already been executed, a ``ptxdist clean root`` followed by a ``ptxdist go`` or
+``ptxdist images`` might be needed to install all debug symbols.)
+
+Due to the size of the resulting root file system, the target root file system
+is often supplied via :ref:`NFSROOT <nfsroot>`.
+
+On the target, GDB can then be used normally (with the caveat that the source files are
+not available unless supplied manually)::
+
+   root@target $ gdb /usr/bin/systemctl 
+   Reading symbols from /usr/bin/systemctl...
+   Reading symbols from /usr/lib/debug/.build-id/fb/3b770e637b51276dcdc41a1e8b37df8a6d6686.debug...
+   (gdb) break main
+   Breakpoint 1 at 0x6c90: file ../systemd-256.1/src/systemctl/systemctl.c, line 1372.
+   (gdb) run
+   Starting program: /usr/bin/systemctl 
+   [Thread debugging using libthread_db enabled]
+   Using host libthread_db library "/lib/libthread_db.so.1".
+
+   Breakpoint 1, main (argc=1, argv=0xbee9bd64)
+       at ../systemd-256.1/src/systemctl/systemctl.c:1372
+   1372    in ../systemd-256.1/src/systemctl/systemctl.c
+   (gdb) backtrace
+   #0  main (argc=1, argv=0xbee9bd64)
+       at ../systemd-256.1/src/systemctl/systemctl.c:1372
+   (gdb) list
+   1372    ../systemd-256.1/src/systemctl/systemctl.c: No such file or directory.
+   (gdb) quit
+
+.. _gdb_via_qemu:
+
+Debugging with CPU Emulation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If we do not need target-specific features to run our application, we can also
+debug it through CPU emulation.
+Two consoles are required for this debug session in this example. We start
+the QEMU in the first console as::
+
+    $ cd platform-nnn/root
+    platform-nnn/root$ qemu-<architecture> -g 1234 -cpu <cpu-core> -L . usr/bin/myapp
 
 The added *-g 1234* parameter lets QEMU wait for a GDB connection to run
 the application.
 
 In the second console we start GDB with the correct architecture
 support. This GDB comes with the same OSELAS.Toolchain that was also
-used to build the project:
-
-.. code-block:: text
+used to build the project::
 
-    $ ./selected_toolchain/<target>-gdb --tui platform-<platformname>/root/usr/bin/myapp
-
-This will run a *curses* based GDB. Not so easy to handle (we must enter
-all the commands and cannot click with a mouse!), but very fast to take
-a quick look at our application.
+    $ ./selected_toolchain/<target>-gdb platform-<platformname>/root/usr/bin/myapp
 
 At first we tell GDB where to look for debug symbols. The correct
-directory here is ``root/``.
-
-.. code-block:: text
+directory here is ``platform-nnn/root/``::
 
     (gdb) set solib-absolute-prefix platform-<platformname>/root
 
-Next we connect this GDB to the waiting QEMU:
-
-.. code-block:: text
+Next we connect this GDB to the waiting QEMU::
 
     (gdb) target remote localhost:1234
     Remote debugging using localhost:1234
     [New Thread 1]
     0x40096a7c in _start () from root/lib/ld.so.1
 
-As our application is already started, we can’t use the GDB command
+As our application is already started via QEMU, we can’t use the GDB command
 ``start`` to run it until it reaches ``main()``. We set a breakpoint
 instead at ``main()`` and *continue* the application:
 
@@ -387,10 +453,6 @@ instead at ``main()`` and *continue* the application:
     Continuing.
     Breakpoint 1, main (argc=1, argv=0x4007f03c) at myapp.c:644
 
-The top part of the running gdbtui console will always show us the
-current source line. Due to the ``root/`` directory usage all
-debug information for GDB is available.
-
 Now we can step through our application by using the commands *step*,
 *next*, *stepi*, *nexti*, *until* and so on.
 
@@ -403,6 +465,64 @@ Now we can step through our application by using the commands *step*,
   debug symbols kept, it will be also possible for GDB to debug C library
   functions our application calls (so it might worth the disk space).
 
+.. _ptxdist_gdb:
+
+Remote-Debugging with *gdbserver*
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When debugging with *gdbserver*, the debugging session is split into two parts:
+the server runs on the target and is remote-controlled by a GDB instance
+running on the build host over a network connection.
+You need to enable the ``PTXCONF_GDBSERVER`` symbol in PTXdists's *menuconfig*
+which makes *gdbserver* available on your target.
+
+Like for the approach above, you will need two terminals.
+In the first terminal you connect to your running target (e.g. via SSH or
+serial console),  start *gdbserver* and tell it the binary to debug and where
+to listen for connections::
+
+   root@target $ gdbserver 192.168.1.11:1234 /usr/bin/systemctl
+   Process /usr/bin/systemctl created; pid = 477
+   Listening on port 1234
+
+In the above example, 192.168.1.11 is the outside-facing IP address of the
+target, and 1234 is the port on which gdbserver listens for connections.
+
+In the other terminal, you then start ``ptxdist gdb``, which will set up GDB
+correctly so that it can find all necessary debug symbols in PTXdist's
+``platform-nnn/root/`` folder, and the sources in PTXdist's
+``platform-nnn/sysroot-target/```.
+You then connect to the *gdbserver* running on the target with GDB's *target
+remote* command::
+
+   user@buildhost $ $ ptxdist gdb 
+   (gdb) target remote 192.168.9.10:1234
+   Remote debugging using 192.168.9.10:1234
+   Reading symbols from …/platform-nnn/root/usr/bin/systemctl...
+   Reading symbols from …/platform-nnn/root/usr/lib/debug/.build-id/fb/3b770e637b51276dcdc41a1e8b37df8a6d6686.debug...
+   Reading symbols from …/platform-nnn/root/lib/ld-linux-armhf.so.3...
+   Reading symbols from …/platform-nnn/root/usr/lib/debug/.build-id/0d/a8d3736dcd31fbf43ea0cb79d1f9699c6695ff.debug...
+   0xb6f29624 in _start () from …/platform-nnn/root/lib/ld-linux-armhf.so.3
+   (gdb) break main
+   Breakpoint 1 at 0x4e3c90: file ../systemd-256.1/src/systemctl/systemctl.c, line 1372.
+   (gdb) continue
+   Continuing.
+
+   Breakpoint 1, main (argc=1, argv=0xbe829d84) at ../systemd-256.1/src/systemctl/systemctl.c:1372
+   1372    DEFINE_MAIN_FUNCTION_WITH_POSITIVE_FAILURE(run);
+   (gdb) list
+   1367    
+   1368            /* Note that we return r here, not 0, so that we can implement the LSB-like return codes */
+   1369            return r;
+   1370    }
+   1371    
+   1372    DEFINE_MAIN_FUNCTION_WITH_POSITIVE_FAILURE(run);
+   1373    #endif
+   (gdb)
+
+You can now use GDB as usual, and when the debugged process terminates (or you
+quit the GDB), *gdbserver* will terminate as well, and close the connection.
+
 Migration between Releases
 --------------------------
 
-- 
2.39.2

[ptxdist] [PATCH v2] doc: daily work: update debugging section

[Roland Hieber]

* Split off the "Running QEMU" section into a stand-alone section
* Mention /usr/lib/debug instead of the old ./debug/ folders
* Describe `ptxdist gdb` and debugging on the target directly
* Keep GDB usage short; refer to third-party documentation
* Use literal blocks instead of code blocks without highlighting
* Make use of the |ptxdistPlatformDir| template variable

Signed-off-by: Roland Hieber <rhi@pengutronix.de>
---
PATCH v2:
* somehow I forgot to stage & amend various changes before sending v1,
  please disregard the old one


 doc/daily_work.inc | 226 ++++++++++++++++++++++++++++++++++-----------
 1 file changed, 174 insertions(+), 52 deletions(-)

diff --git a/doc/daily_work.inc b/doc/daily_work.inc
index b98adf75197d..65b0bad25d97 100644
--- a/doc/daily_work.inc
+++ b/doc/daily_work.inc
@@ -285,15 +285,11 @@ nonexisting file, we can limit the output to all ``open`` system calls:
 The binary may fail due to a missing ``/etc/foo.conf``. This could be a
 hint on what is going wrong (it might not be the final solution).
 
-Debugging with CPU emulation
-----------------------------
-
-If we do not need some target related feature to run our application, we
-can also debug it through a simple CPU emulation. Thanks to QEMU we can
-run ELF binaries for other architectures than our build host is.
+Running an Application Made for a Different Architecture
+--------------------------------------------------------
 
-Running an Application made for a different Architecture
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Thanks to QEMU we can run ELF binaries for different architectures than our
+build host.
 
 PTXdist creates a fully working root filesystem with all run-time
 components in ``root/``. Lets assume we made a PTXdist based project for
@@ -318,64 +314,134 @@ host’s operating system. Using QEMU in this way let us simply check our
 programs. There are also QEMU environments for other architectures
 available.
 
-Debugging an Application made for a different Architecture
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Debugging our application is also possible with QEMU. All we need are a
-root filesystem with debug symbols available, QEMU and an architecture
-aware debugger.
-
-The root filesystem with debug symbols will be provided by PTXdist, the
-architecture aware debugger comes with the OSELAS.Toolchain. Two
-consoles are required for this debug session in this example. We start
-the QEMU in the first console as:
-
-.. code-block:: text
-
-    $ cd ptxdistPlatformDir/root
-    ptxdistPlatformDir/root$ qemu-<architecture> -g 1234 -cpu <cpu-core> -L . usr/bin/myapp
+Debugging an Application Made for a Different Architecture
+----------------------------------------------------------
+
+Debugging an application is possible in three ways:
+
+#. :ref:`Directly running GDB on the target hardware <gdb_on_target>`:
+   This approach has the advantage that the target system is "self-contained",
+   but it is necessary to install the debug symbols in the target's root file
+   system, which can significantly improve the memory footprint. Furthermore,
+   it is often not possible for GDB to access the application's source code
+   unless further steps are taken (e.g. making the source code available in the
+   correct paths over an NFS share or from a USB drive).
+
+#. :ref:`Running the application in a QEMU emulating the target architecture <gdb_via_qemu>`:
+   This approach doesn't require any additional hardware and can therefore be
+   done completely on the build host, but it is limited to applications
+   that do not depend on specific behaviour of the target hardware, like
+   using external interfaces or architecture-specific features that are not
+   available via QEMU (e.g. accessing GPS hardware over UART, or graphics
+   acceleration).
+
+#. :ref:`Running gdbserver on the target hardware, and connecting to it with GDB
+   from the build host <ptxdist_gdb>`:
+   This is the simplest use case when a target hardware with network interfaces
+   is available, and PTXdist includes native support for it.
+   In this case, the target file system only has to include the *gdbserver*
+   binary, but does not need to contain any further debug information or source
+   code, which are supplied by the build host.
+
+In any case, we need the respective binaries including their debug information,
+and a debugger for the target architecture.
+The root filesystem with debug symbols is provided by PTXdist in
+``|ptxdistPlatformDir|/root/`` on the build host, and the architecture-aware debugger
+comes with the OSELAS.Toolchain.
+
+.. note:: When PTXdist builds a target root file system, the binaries that are
+   installed into it during the *targetinstall* stage of each package are
+   always stripped of their debug symbols.  Instead, the debug symbols are
+   installed under ``|ptxdistPlatformDir|/root/usr/lib/debug/`` in separate files named
+   after the *build ID* of the binary (which is a checksum over the contents of
+   the binary, and can change when a binary is rebuilt).
+
+   When GDB loads a binary, it will look for and automatically load matching
+   debug symbols in ``usr/lib/debug`` relative to its *sysroot*, which is
+   usually ``/``, but can be configured at runtime.
+
+.. note:: Information on using GDB is not part of this manual.
+   There are countless cheatsheets available on the internet, but in most
+   cases you will get around sufficiently with the ``break <location|function>``,
+   ``run``, ``next``, ``step``, ``finish``, ``continue``, ``print <statement|address>``,
+   ``list <location|function>`` and ``quit`` commands.
+   Current GDB versions also include a well-structured online help with the
+   ``help`` command, which can show you how to use each command.
+
+.. _gdb_on_target:
+
+Running GDB on the Target Hardware
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In this case, GDB and the debug information need to be available on the target hardware.
+GDB can be enabled in the *menuconfig* with the ``PTXCONF_GDB`` symbol.
+Additionally, the debug symbols are installed into the target root file system 
+when the ``PTXCONF_DEBUG_PACKAGES_INSTALL`` option is enabled.
+(If an image has been built before, i.e. the *targetinstall* stages have
+already been executed, a ``ptxdist clean root`` followed by a ``ptxdist go`` or
+``ptxdist images`` might be needed to install all debug symbols.)
+
+Due to the size of the resulting root file system, the target root file system
+is often supplied via :ref:`NFSROOT <nfsroot>`.
+
+On the target, GDB can then be used normally (with the caveat that the source files are
+not available unless supplied manually)::
+
+   root@target $ gdb /usr/bin/systemctl 
+   Reading symbols from /usr/bin/systemctl...
+   Reading symbols from /usr/lib/debug/.build-id/fb/3b770e637b51276dcdc41a1e8b37df8a6d6686.debug...
+   (gdb) break main
+   Breakpoint 1 at 0x6c90: file ../systemd-256.1/src/systemctl/systemctl.c, line 1372.
+   (gdb) run
+   Starting program: /usr/bin/systemctl 
+   [Thread debugging using libthread_db enabled]
+   Using host libthread_db library "/lib/libthread_db.so.1".
+
+   Breakpoint 1, main (argc=1, argv=0xbee9bd64)
+       at ../systemd-256.1/src/systemctl/systemctl.c:1372
+   1372    in ../systemd-256.1/src/systemctl/systemctl.c
+   (gdb) backtrace
+   #0  main (argc=1, argv=0xbee9bd64)
+       at ../systemd-256.1/src/systemctl/systemctl.c:1372
+   (gdb) list
+   1372    ../systemd-256.1/src/systemctl/systemctl.c: No such file or directory.
+   (gdb) quit
+
+.. _gdb_via_qemu:
+
+Debugging with CPU Emulation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If we do not need target-specific features to run our application, we can also
+debug it through CPU emulation.
+Two consoles are required for this debug session in this example. We start
+the QEMU in the first console as::
 
-.. note:: PTXdist always builds a root filesystem ``root/``.
-  It contains all components without debug 
-  information (all binaries are in the same size as used later on on the
-  real target). In addition, each directory that contains binaries also
-  contains a ``.debug/`` directory. It contains a file with only the debug
-  symbols for each binary. These files are ignored while running
-  applications but GDB knows about it and will automatically load the debug
-  files.
+    $ cd |ptxdistPlatformDir|/root
+    |ptxdistPlatformDir|/root$ qemu-<architecture> -g 1234 -cpu <cpu-core> -L . usr/bin/myapp
 
 The added *-g 1234* parameter lets QEMU wait for a GDB connection to run
 the application.
 
 In the second console we start GDB with the correct architecture
 support. This GDB comes with the same OSELAS.Toolchain that was also
-used to build the project:
-
-.. code-block:: text
+used to build the project::
 
-    $ ./selected_toolchain/<target>-gdb --tui platform-<platformname>/root/usr/bin/myapp
-
-This will run a *curses* based GDB. Not so easy to handle (we must enter
-all the commands and cannot click with a mouse!), but very fast to take
-a quick look at our application.
+    $ ./selected_toolchain/<target>-gdb |ptxdistPlatformDir|/root/usr/bin/myapp
 
 At first we tell GDB where to look for debug symbols. The correct
-directory here is ``root/``.
-
-.. code-block:: text
+directory here is ``|ptxdistPlatformDir|/root/``::
 
-    (gdb) set solib-absolute-prefix platform-<platformname>/root
+    (gdb) set solib-absolute-prefix |ptxdistPlatformDir|/root
 
-Next we connect this GDB to the waiting QEMU:
-
-.. code-block:: text
+Next we connect this GDB to the waiting QEMU::
 
     (gdb) target remote localhost:1234
     Remote debugging using localhost:1234
     [New Thread 1]
     0x40096a7c in _start () from root/lib/ld.so.1
 
-As our application is already started, we can’t use the GDB command
+As our application is already started via QEMU, we can’t use the GDB command
 ``start`` to run it until it reaches ``main()``. We set a breakpoint
 instead at ``main()`` and *continue* the application:
 
@@ -387,10 +453,6 @@ instead at ``main()`` and *continue* the application:
     Continuing.
     Breakpoint 1, main (argc=1, argv=0x4007f03c) at myapp.c:644
 
-The top part of the running gdbtui console will always show us the
-current source line. Due to the ``root/`` directory usage all
-debug information for GDB is available.
-
 Now we can step through our application by using the commands *step*,
 *next*, *stepi*, *nexti*, *until* and so on.
 
@@ -403,6 +465,66 @@ Now we can step through our application by using the commands *step*,
   debug symbols kept, it will be also possible for GDB to debug C library
   functions our application calls (so it might worth the disk space).
 
+.. _ptxdist_gdb:
+
+Remote-Debugging with *gdbserver*
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When debugging with *gdbserver*, the debugging session is split into two parts:
+the server runs on the target and is remote-controlled by a GDB instance
+running on the build host over a network connection.
+You need to enable the ``PTXCONF_GDBSERVER`` symbol in PTXdists's *menuconfig*
+which makes *gdbserver* available on your target.
+
+Like for the approach above, you will need two terminals.
+In the first terminal you connect to your running target (e.g. via SSH or
+serial console), start *gdbserver* and tell it the binary to debug and where
+to listen for connections::
+
+   root@target $ gdbserver 192.168.1.11:1234 /usr/bin/systemctl
+   Process /usr/bin/systemctl created; pid = 477
+   Listening on port 1234
+
+In the above example, 192.168.1.11 is the outside-facing IP address of the
+target, and 1234 is the port on which to listen.
+gdbserver has now created a process, and waits for connections from a GDB.
+
+In the other terminal, you then start ``ptxdist gdb``, which will set up GDB
+correctly so that it can find all necessary debug symbols in PTXdist's
+``|ptxdistPlatformDir|/root/`` folder, and the sources in
+``|ptxdistPlatformDir|/sysroot-target/``.
+You then connect to the gdbserver instance running on the target using GDB's
+*target remote* command::
+
+   user@buildhost $ ptxdist gdb
+   (gdb) target remote 192.168.9.10:1234
+   Remote debugging using 192.168.9.10:1234
+   Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/bin/systemctl...
+   Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/lib/debug/.build-id/fb/3b770e637b51276dcdc41a1e8b37df8a6d6686.debug...
+   Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/lib/ld-linux-armhf.so.3...
+   Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/lib/debug/.build-id/0d/a8d3736dcd31fbf43ea0cb79d1f9699c6695ff.debug...
+   0xb6f29624 in _start () from …/my-bsp/|ptxdistPlatformDir|/root/lib/ld-linux-armhf.so.3
+   (gdb) break main
+   Breakpoint 1 at 0x4e3c90: file ../systemd-256.1/src/systemctl/systemctl.c, line 1372.
+   (gdb) continue
+   Continuing.
+
+   Breakpoint 1, main (argc=1, argv=0xbe829d84) at ../systemd-256.1/src/systemctl/systemctl.c:1372
+   1372    DEFINE_MAIN_FUNCTION_WITH_POSITIVE_FAILURE(run);
+   (gdb) list
+   1367    
+   1368            /* Note that we return r here, not 0, so that we can implement the LSB-like return codes */
+   1369            return r;
+   1370    }
+   1371    
+   1372    DEFINE_MAIN_FUNCTION_WITH_POSITIVE_FAILURE(run);
+   1373    #endif
+   (gdb)
+
+You can now use GDB as usual, and when the debugged process terminates (or you
+quit the GDB session), *gdbserver* will terminate as well, and close the
+connection.
+
 Migration between Releases
 --------------------------
 
-- 
2.39.2

Re: [ptxdist] [PATCH v2] doc: daily work: update debugging section

[Michael Olbrich]

On Mon, Aug 26, 2024 at 02:04:18PM +0200, Roland Hieber wrote:
> * Split off the "Running QEMU" section into a stand-alone section
> * Mention /usr/lib/debug instead of the old ./debug/ folders
> * Describe `ptxdist gdb` and debugging on the target directly
> * Keep GDB usage short; refer to third-party documentation
> * Use literal blocks instead of code blocks without highlighting
> * Make use of the |ptxdistPlatformDir| template variable
> 
> Signed-off-by: Roland Hieber <rhi@pengutronix.de>
> ---
> PATCH v2:
> * somehow I forgot to stage & amend various changes before sending v1,
>   please disregard the old one
> 
> 
>  doc/daily_work.inc | 226 ++++++++++++++++++++++++++++++++++-----------
>  1 file changed, 174 insertions(+), 52 deletions(-)
> 
> diff --git a/doc/daily_work.inc b/doc/daily_work.inc
> index b98adf75197d..65b0bad25d97 100644
> --- a/doc/daily_work.inc
> +++ b/doc/daily_work.inc
> @@ -285,15 +285,11 @@ nonexisting file, we can limit the output to all ``open`` system calls:
>  The binary may fail due to a missing ``/etc/foo.conf``. This could be a
>  hint on what is going wrong (it might not be the final solution).
>  
> -Debugging with CPU emulation
> -----------------------------
> -
> -If we do not need some target related feature to run our application, we
> -can also debug it through a simple CPU emulation. Thanks to QEMU we can
> -run ELF binaries for other architectures than our build host is.
> +Running an Application Made for a Different Architecture
> +--------------------------------------------------------
>  
> -Running an Application made for a different Architecture
> -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +Thanks to QEMU we can run ELF binaries for different architectures than our
> +build host.
>  
>  PTXdist creates a fully working root filesystem with all run-time
>  components in ``root/``. Lets assume we made a PTXdist based project for
> @@ -318,64 +314,134 @@ host’s operating system. Using QEMU in this way let us simply check our
>  programs. There are also QEMU environments for other architectures
>  available.
>  
> -Debugging an Application made for a different Architecture
> -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> -
> -Debugging our application is also possible with QEMU. All we need are a
> -root filesystem with debug symbols available, QEMU and an architecture
> -aware debugger.
> -
> -The root filesystem with debug symbols will be provided by PTXdist, the
> -architecture aware debugger comes with the OSELAS.Toolchain. Two
> -consoles are required for this debug session in this example. We start
> -the QEMU in the first console as:
> -
> -.. code-block:: text
> -
> -    $ cd ptxdistPlatformDir/root
> -    ptxdistPlatformDir/root$ qemu-<architecture> -g 1234 -cpu <cpu-core> -L . usr/bin/myapp
> +Debugging an Application Made for a Different Architecture
> +----------------------------------------------------------
> +
> +Debugging an application is possible in three ways:
> +
> +#. :ref:`Directly running GDB on the target hardware <gdb_on_target>`:
> +   This approach has the advantage that the target system is "self-contained",
> +   but it is necessary to install the debug symbols in the target's root file
> +   system, which can significantly improve the memory footprint. Furthermore,

"improve" sounds wrong here. Maybe you meant "increase"?

> +   it is often not possible for GDB to access the application's source code
> +   unless further steps are taken (e.g. making the source code available in the
> +   correct paths over an NFS share or from a USB drive).
> +
> +#. :ref:`Running the application in a QEMU emulating the target architecture <gdb_via_qemu>`:
> +   This approach doesn't require any additional hardware and can therefore be
> +   done completely on the build host, but it is limited to applications
> +   that do not depend on specific behaviour of the target hardware, like
> +   using external interfaces or architecture-specific features that are not
> +   available via QEMU (e.g. accessing GPS hardware over UART, or graphics
> +   acceleration).
> +
> +#. :ref:`Running gdbserver on the target hardware, and connecting to it with GDB
> +   from the build host <ptxdist_gdb>`:
> +   This is the simplest use case when a target hardware with network interfaces
> +   is available, and PTXdist includes native support for it.
> +   In this case, the target file system only has to include the *gdbserver*
> +   binary, but does not need to contain any further debug information or source
> +   code, which are supplied by the build host.
> +
> +In any case, we need the respective binaries including their debug information,
> +and a debugger for the target architecture.
> +The root filesystem with debug symbols is provided by PTXdist in
> +``|ptxdistPlatformDir|/root/`` on the build host, and the architecture-aware debugger
> +comes with the OSELAS.Toolchain.
> +
> +.. note:: When PTXdist builds a target root file system, the binaries that are
> +   installed into it during the *targetinstall* stage of each package are
> +   always stripped of their debug symbols.  Instead, the debug symbols are
> +   installed under ``|ptxdistPlatformDir|/root/usr/lib/debug/`` in separate files named
> +   after the *build ID* of the binary (which is a checksum over the contents of
> +   the binary, and can change when a binary is rebuilt).
> +
> +   When GDB loads a binary, it will look for and automatically load matching
> +   debug symbols in ``usr/lib/debug`` relative to its *sysroot*, which is
> +   usually ``/``, but can be configured at runtime.
> +
> +.. note:: Information on using GDB is not part of this manual.
> +   There are countless cheatsheets available on the internet, but in most
> +   cases you will get around sufficiently with the ``break <location|function>``,
> +   ``run``, ``next``, ``step``, ``finish``, ``continue``, ``print <statement|address>``,
> +   ``list <location|function>`` and ``quit`` commands.
> +   Current GDB versions also include a well-structured online help with the
> +   ``help`` command, which can show you how to use each command.
> +
> +.. _gdb_on_target:
> +
> +Running GDB on the Target Hardware
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +In this case, GDB and the debug information need to be available on the target hardware.
> +GDB can be enabled in the *menuconfig* with the ``PTXCONF_GDB`` symbol.
> +Additionally, the debug symbols are installed into the target root file system 
> +when the ``PTXCONF_DEBUG_PACKAGES_INSTALL`` option is enabled.
> +(If an image has been built before, i.e. the *targetinstall* stages have
> +already been executed, a ``ptxdist clean root`` followed by a ``ptxdist go`` or
> +``ptxdist images`` might be needed to install all debug symbols.)
> +
> +Due to the size of the resulting root file system, the target root file system
> +is often supplied via :ref:`NFSROOT <nfsroot>`.
> +
> +On the target, GDB can then be used normally (with the caveat that the source files are
> +not available unless supplied manually)::
> +
> +   root@target $ gdb /usr/bin/systemctl 
> +   Reading symbols from /usr/bin/systemctl...
> +   Reading symbols from /usr/lib/debug/.build-id/fb/3b770e637b51276dcdc41a1e8b37df8a6d6686.debug...
> +   (gdb) break main
> +   Breakpoint 1 at 0x6c90: file ../systemd-256.1/src/systemctl/systemctl.c, line 1372.
> +   (gdb) run
> +   Starting program: /usr/bin/systemctl 
> +   [Thread debugging using libthread_db enabled]
> +   Using host libthread_db library "/lib/libthread_db.so.1".
> +
> +   Breakpoint 1, main (argc=1, argv=0xbee9bd64)
> +       at ../systemd-256.1/src/systemctl/systemctl.c:1372
> +   1372    in ../systemd-256.1/src/systemctl/systemctl.c
> +   (gdb) backtrace
> +   #0  main (argc=1, argv=0xbee9bd64)
> +       at ../systemd-256.1/src/systemctl/systemctl.c:1372
> +   (gdb) list
> +   1372    ../systemd-256.1/src/systemctl/systemctl.c: No such file or directory.
> +   (gdb) quit
> +
> +.. _gdb_via_qemu:
> +
> +Debugging with CPU Emulation
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +If we do not need target-specific features to run our application, we can also
> +debug it through CPU emulation.
> +Two consoles are required for this debug session in this example. We start
> +the QEMU in the first console as::
>  
> -.. note:: PTXdist always builds a root filesystem ``root/``.
> -  It contains all components without debug 
> -  information (all binaries are in the same size as used later on on the
> -  real target). In addition, each directory that contains binaries also
> -  contains a ``.debug/`` directory. It contains a file with only the debug
> -  symbols for each binary. These files are ignored while running
> -  applications but GDB knows about it and will automatically load the debug
> -  files.
> +    $ cd |ptxdistPlatformDir|/root
> +    |ptxdistPlatformDir|/root$ qemu-<architecture> -g 1234 -cpu <cpu-core> -L . usr/bin/myapp
>  
>  The added *-g 1234* parameter lets QEMU wait for a GDB connection to run
>  the application.
>  
>  In the second console we start GDB with the correct architecture
>  support. This GDB comes with the same OSELAS.Toolchain that was also
> -used to build the project:
> -
> -.. code-block:: text
> +used to build the project::
>  
> -    $ ./selected_toolchain/<target>-gdb --tui platform-<platformname>/root/usr/bin/myapp
> -
> -This will run a *curses* based GDB. Not so easy to handle (we must enter
> -all the commands and cannot click with a mouse!), but very fast to take
> -a quick look at our application.
> +    $ ./selected_toolchain/<target>-gdb |ptxdistPlatformDir|/root/usr/bin/myapp
>  
>  At first we tell GDB where to look for debug symbols. The correct
> -directory here is ``root/``.
> -
> -.. code-block:: text
> +directory here is ``|ptxdistPlatformDir|/root/``::
>  
> -    (gdb) set solib-absolute-prefix platform-<platformname>/root
> +    (gdb) set solib-absolute-prefix |ptxdistPlatformDir|/root

This part of the documentation is pretty old and a bit outdated. Can you
change it to use 'ptxdist gdb'? It should possible to use it in this case
and then manually setting 'solib-absolute-prefix' is not needed.

Michael

>  
> -Next we connect this GDB to the waiting QEMU:
> -
> -.. code-block:: text
> +Next we connect this GDB to the waiting QEMU::
>  
>      (gdb) target remote localhost:1234
>      Remote debugging using localhost:1234
>      [New Thread 1]
>      0x40096a7c in _start () from root/lib/ld.so.1
>  
> -As our application is already started, we can’t use the GDB command
> +As our application is already started via QEMU, we can’t use the GDB command
>  ``start`` to run it until it reaches ``main()``. We set a breakpoint
>  instead at ``main()`` and *continue* the application:
>  
> @@ -387,10 +453,6 @@ instead at ``main()`` and *continue* the application:
>      Continuing.
>      Breakpoint 1, main (argc=1, argv=0x4007f03c) at myapp.c:644
>  
> -The top part of the running gdbtui console will always show us the
> -current source line. Due to the ``root/`` directory usage all
> -debug information for GDB is available.
> -
>  Now we can step through our application by using the commands *step*,
>  *next*, *stepi*, *nexti*, *until* and so on.
>  
> @@ -403,6 +465,66 @@ Now we can step through our application by using the commands *step*,
>    debug symbols kept, it will be also possible for GDB to debug C library
>    functions our application calls (so it might worth the disk space).
>  
> +.. _ptxdist_gdb:
> +
> +Remote-Debugging with *gdbserver*
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +When debugging with *gdbserver*, the debugging session is split into two parts:
> +the server runs on the target and is remote-controlled by a GDB instance
> +running on the build host over a network connection.
> +You need to enable the ``PTXCONF_GDBSERVER`` symbol in PTXdists's *menuconfig*
> +which makes *gdbserver* available on your target.
> +
> +Like for the approach above, you will need two terminals.
> +In the first terminal you connect to your running target (e.g. via SSH or
> +serial console), start *gdbserver* and tell it the binary to debug and where
> +to listen for connections::
> +
> +   root@target $ gdbserver 192.168.1.11:1234 /usr/bin/systemctl
> +   Process /usr/bin/systemctl created; pid = 477
> +   Listening on port 1234
> +
> +In the above example, 192.168.1.11 is the outside-facing IP address of the
> +target, and 1234 is the port on which to listen.
> +gdbserver has now created a process, and waits for connections from a GDB.
> +
> +In the other terminal, you then start ``ptxdist gdb``, which will set up GDB
> +correctly so that it can find all necessary debug symbols in PTXdist's
> +``|ptxdistPlatformDir|/root/`` folder, and the sources in
> +``|ptxdistPlatformDir|/sysroot-target/``.
> +You then connect to the gdbserver instance running on the target using GDB's
> +*target remote* command::
> +
> +   user@buildhost $ ptxdist gdb
> +   (gdb) target remote 192.168.9.10:1234
> +   Remote debugging using 192.168.9.10:1234
> +   Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/bin/systemctl...
> +   Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/lib/debug/.build-id/fb/3b770e637b51276dcdc41a1e8b37df8a6d6686.debug...
> +   Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/lib/ld-linux-armhf.so.3...
> +   Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/lib/debug/.build-id/0d/a8d3736dcd31fbf43ea0cb79d1f9699c6695ff.debug...
> +   0xb6f29624 in _start () from …/my-bsp/|ptxdistPlatformDir|/root/lib/ld-linux-armhf.so.3
> +   (gdb) break main
> +   Breakpoint 1 at 0x4e3c90: file ../systemd-256.1/src/systemctl/systemctl.c, line 1372.
> +   (gdb) continue
> +   Continuing.
> +
> +   Breakpoint 1, main (argc=1, argv=0xbe829d84) at ../systemd-256.1/src/systemctl/systemctl.c:1372
> +   1372    DEFINE_MAIN_FUNCTION_WITH_POSITIVE_FAILURE(run);
> +   (gdb) list
> +   1367    
> +   1368            /* Note that we return r here, not 0, so that we can implement the LSB-like return codes */
> +   1369            return r;
> +   1370    }
> +   1371    
> +   1372    DEFINE_MAIN_FUNCTION_WITH_POSITIVE_FAILURE(run);
> +   1373    #endif
> +   (gdb)
> +
> +You can now use GDB as usual, and when the debugged process terminates (or you
> +quit the GDB session), *gdbserver* will terminate as well, and close the
> +connection.
> +
>  Migration between Releases
>  --------------------------
>  
> -- 
> 2.39.2
> 
> 
> 

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |