From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Mon, 26 Aug 2024 14:04:39 +0200 Received: from metis.whiteo.stw.pengutronix.de ([2a0a:edc0:2:b01:1d::104]) by lore.white.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1siYSd-00DZxI-1t for lore@lore.pengutronix.de; Mon, 26 Aug 2024 14:04:39 +0200 Received: from localhost ([127.0.0.1] helo=metis.whiteo.stw.pengutronix.de) by metis.whiteo.stw.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1siYSd-0002Ty-1c; Mon, 26 Aug 2024 14:04:39 +0200 Received: from drehscheibe.grey.stw.pengutronix.de ([2a0a:edc0:0:c01:1d::a2]) by metis.whiteo.stw.pengutronix.de with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1siYSK-0002Td-02; Mon, 26 Aug 2024 14:04:20 +0200 Received: from [2a0a:edc0:0:1101:1d::ac] (helo=dude04.red.stw.pengutronix.de) by drehscheibe.grey.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1siYSJ-003BGr-J4; Mon, 26 Aug 2024 14:04:19 +0200 Received: from rhi by dude04.red.stw.pengutronix.de with local (Exim 4.96) (envelope-from ) id 1siYSJ-00AbqJ-1k; Mon, 26 Aug 2024 14:04:19 +0200 From: Roland Hieber To: ptxdist@pengutronix.de Date: Mon, 26 Aug 2024 14:04:18 +0200 Message-Id: <20240826120418.2528432-1-rhi@pengutronix.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240823152620.954595-1-rhi@pengutronix.de> References: <20240823152620.954595-1-rhi@pengutronix.de> MIME-Version: 1.0 Mail-Followup-To: Roland Hieber , ptxdist@pengutronix.de Content-Type: text/plain; charset=utf8 Content-Transfer-Encoding: 8bit Subject: [ptxdist] [PATCH v2] doc: daily work: update debugging section X-BeenThere: ptxdist@pengutronix.de X-Mailman-Version: 2.1.29 Precedence: list List-Id: PTXdist Development Mailing List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ptxdist@pengutronix.de Cc: Roland Hieber Sender: "ptxdist" X-SA-Exim-Connect-IP: 127.0.0.1 X-SA-Exim-Mail-From: ptxdist-bounces@pengutronix.de X-SA-Exim-Scanned: No (on metis.whiteo.stw.pengutronix.de); SAEximRunCond expanded to false * Split off the "Running QEMU" section into a stand-alone section * Mention /usr/lib/debug instead of the old ./debug/ folders * Describe `ptxdist gdb` and debugging on the target directly * Keep GDB usage short; refer to third-party documentation * Use literal blocks instead of code blocks without highlighting * Make use of the |ptxdistPlatformDir| template variable Signed-off-by: Roland Hieber --- PATCH v2: * somehow I forgot to stage & amend various changes before sending v1, please disregard the old one doc/daily_work.inc | 226 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 174 insertions(+), 52 deletions(-) diff --git a/doc/daily_work.inc b/doc/daily_work.inc index b98adf75197d..65b0bad25d97 100644 --- a/doc/daily_work.inc +++ b/doc/daily_work.inc @@ -285,15 +285,11 @@ nonexisting file, we can limit the output to all ``open`` system calls: The binary may fail due to a missing ``/etc/foo.conf``. This could be a hint on what is going wrong (it might not be the final solution). -Debugging with CPU emulation ----------------------------- - -If we do not need some target related feature to run our application, we -can also debug it through a simple CPU emulation. Thanks to QEMU we can -run ELF binaries for other architectures than our build host is. +Running an Application Made for a Different Architecture +-------------------------------------------------------- -Running an Application made for a different Architecture -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Thanks to QEMU we can run ELF binaries for different architectures than our +build host. PTXdist creates a fully working root filesystem with all run-time components in ``root/``. Lets assume we made a PTXdist based project for @@ -318,64 +314,134 @@ host’s operating system. Using QEMU in this way let us simply check our programs. There are also QEMU environments for other architectures available. -Debugging an Application made for a different Architecture -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Debugging our application is also possible with QEMU. All we need are a -root filesystem with debug symbols available, QEMU and an architecture -aware debugger. - -The root filesystem with debug symbols will be provided by PTXdist, the -architecture aware debugger comes with the OSELAS.Toolchain. Two -consoles are required for this debug session in this example. We start -the QEMU in the first console as: - -.. code-block:: text - - $ cd ptxdistPlatformDir/root - ptxdistPlatformDir/root$ qemu- -g 1234 -cpu -L . usr/bin/myapp +Debugging an Application Made for a Different Architecture +---------------------------------------------------------- + +Debugging an application is possible in three ways: + +#. :ref:`Directly running GDB on the target hardware `: + This approach has the advantage that the target system is "self-contained", + but it is necessary to install the debug symbols in the target's root file + system, which can significantly improve the memory footprint. Furthermore, + it is often not possible for GDB to access the application's source code + unless further steps are taken (e.g. making the source code available in the + correct paths over an NFS share or from a USB drive). + +#. :ref:`Running the application in a QEMU emulating the target architecture `: + This approach doesn't require any additional hardware and can therefore be + done completely on the build host, but it is limited to applications + that do not depend on specific behaviour of the target hardware, like + using external interfaces or architecture-specific features that are not + available via QEMU (e.g. accessing GPS hardware over UART, or graphics + acceleration). + +#. :ref:`Running gdbserver on the target hardware, and connecting to it with GDB + from the build host `: + This is the simplest use case when a target hardware with network interfaces + is available, and PTXdist includes native support for it. + In this case, the target file system only has to include the *gdbserver* + binary, but does not need to contain any further debug information or source + code, which are supplied by the build host. + +In any case, we need the respective binaries including their debug information, +and a debugger for the target architecture. +The root filesystem with debug symbols is provided by PTXdist in +``|ptxdistPlatformDir|/root/`` on the build host, and the architecture-aware debugger +comes with the OSELAS.Toolchain. + +.. note:: When PTXdist builds a target root file system, the binaries that are + installed into it during the *targetinstall* stage of each package are + always stripped of their debug symbols. Instead, the debug symbols are + installed under ``|ptxdistPlatformDir|/root/usr/lib/debug/`` in separate files named + after the *build ID* of the binary (which is a checksum over the contents of + the binary, and can change when a binary is rebuilt). + + When GDB loads a binary, it will look for and automatically load matching + debug symbols in ``usr/lib/debug`` relative to its *sysroot*, which is + usually ``/``, but can be configured at runtime. + +.. note:: Information on using GDB is not part of this manual. + There are countless cheatsheets available on the internet, but in most + cases you will get around sufficiently with the ``break ``, + ``run``, ``next``, ``step``, ``finish``, ``continue``, ``print ``, + ``list `` and ``quit`` commands. + Current GDB versions also include a well-structured online help with the + ``help`` command, which can show you how to use each command. + +.. _gdb_on_target: + +Running GDB on the Target Hardware +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In this case, GDB and the debug information need to be available on the target hardware. +GDB can be enabled in the *menuconfig* with the ``PTXCONF_GDB`` symbol. +Additionally, the debug symbols are installed into the target root file system +when the ``PTXCONF_DEBUG_PACKAGES_INSTALL`` option is enabled. +(If an image has been built before, i.e. the *targetinstall* stages have +already been executed, a ``ptxdist clean root`` followed by a ``ptxdist go`` or +``ptxdist images`` might be needed to install all debug symbols.) + +Due to the size of the resulting root file system, the target root file system +is often supplied via :ref:`NFSROOT `. + +On the target, GDB can then be used normally (with the caveat that the source files are +not available unless supplied manually):: + + root@target $ gdb /usr/bin/systemctl + Reading symbols from /usr/bin/systemctl... + Reading symbols from /usr/lib/debug/.build-id/fb/3b770e637b51276dcdc41a1e8b37df8a6d6686.debug... + (gdb) break main + Breakpoint 1 at 0x6c90: file ../systemd-256.1/src/systemctl/systemctl.c, line 1372. + (gdb) run + Starting program: /usr/bin/systemctl + [Thread debugging using libthread_db enabled] + Using host libthread_db library "/lib/libthread_db.so.1". + + Breakpoint 1, main (argc=1, argv=0xbee9bd64) + at ../systemd-256.1/src/systemctl/systemctl.c:1372 + 1372 in ../systemd-256.1/src/systemctl/systemctl.c + (gdb) backtrace + #0 main (argc=1, argv=0xbee9bd64) + at ../systemd-256.1/src/systemctl/systemctl.c:1372 + (gdb) list + 1372 ../systemd-256.1/src/systemctl/systemctl.c: No such file or directory. + (gdb) quit + +.. _gdb_via_qemu: + +Debugging with CPU Emulation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If we do not need target-specific features to run our application, we can also +debug it through CPU emulation. +Two consoles are required for this debug session in this example. We start +the QEMU in the first console as:: -.. note:: PTXdist always builds a root filesystem ``root/``. - It contains all components without debug - information (all binaries are in the same size as used later on on the - real target). In addition, each directory that contains binaries also - contains a ``.debug/`` directory. It contains a file with only the debug - symbols for each binary. These files are ignored while running - applications but GDB knows about it and will automatically load the debug - files. + $ cd |ptxdistPlatformDir|/root + |ptxdistPlatformDir|/root$ qemu- -g 1234 -cpu -L . usr/bin/myapp The added *-g 1234* parameter lets QEMU wait for a GDB connection to run the application. In the second console we start GDB with the correct architecture support. This GDB comes with the same OSELAS.Toolchain that was also -used to build the project: - -.. code-block:: text +used to build the project:: - $ ./selected_toolchain/-gdb --tui platform-/root/usr/bin/myapp - -This will run a *curses* based GDB. Not so easy to handle (we must enter -all the commands and cannot click with a mouse!), but very fast to take -a quick look at our application. + $ ./selected_toolchain/-gdb |ptxdistPlatformDir|/root/usr/bin/myapp At first we tell GDB where to look for debug symbols. The correct -directory here is ``root/``. - -.. code-block:: text +directory here is ``|ptxdistPlatformDir|/root/``:: - (gdb) set solib-absolute-prefix platform-/root + (gdb) set solib-absolute-prefix |ptxdistPlatformDir|/root -Next we connect this GDB to the waiting QEMU: - -.. code-block:: text +Next we connect this GDB to the waiting QEMU:: (gdb) target remote localhost:1234 Remote debugging using localhost:1234 [New Thread 1] 0x40096a7c in _start () from root/lib/ld.so.1 -As our application is already started, we can’t use the GDB command +As our application is already started via QEMU, we can’t use the GDB command ``start`` to run it until it reaches ``main()``. We set a breakpoint instead at ``main()`` and *continue* the application: @@ -387,10 +453,6 @@ instead at ``main()`` and *continue* the application: Continuing. Breakpoint 1, main (argc=1, argv=0x4007f03c) at myapp.c:644 -The top part of the running gdbtui console will always show us the -current source line. Due to the ``root/`` directory usage all -debug information for GDB is available. - Now we can step through our application by using the commands *step*, *next*, *stepi*, *nexti*, *until* and so on. @@ -403,6 +465,66 @@ Now we can step through our application by using the commands *step*, debug symbols kept, it will be also possible for GDB to debug C library functions our application calls (so it might worth the disk space). +.. _ptxdist_gdb: + +Remote-Debugging with *gdbserver* +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When debugging with *gdbserver*, the debugging session is split into two parts: +the server runs on the target and is remote-controlled by a GDB instance +running on the build host over a network connection. +You need to enable the ``PTXCONF_GDBSERVER`` symbol in PTXdists's *menuconfig* +which makes *gdbserver* available on your target. + +Like for the approach above, you will need two terminals. +In the first terminal you connect to your running target (e.g. via SSH or +serial console), start *gdbserver* and tell it the binary to debug and where +to listen for connections:: + + root@target $ gdbserver 192.168.1.11:1234 /usr/bin/systemctl + Process /usr/bin/systemctl created; pid = 477 + Listening on port 1234 + +In the above example, 192.168.1.11 is the outside-facing IP address of the +target, and 1234 is the port on which to listen. +gdbserver has now created a process, and waits for connections from a GDB. + +In the other terminal, you then start ``ptxdist gdb``, which will set up GDB +correctly so that it can find all necessary debug symbols in PTXdist's +``|ptxdistPlatformDir|/root/`` folder, and the sources in +``|ptxdistPlatformDir|/sysroot-target/``. +You then connect to the gdbserver instance running on the target using GDB's +*target remote* command:: + + user@buildhost $ ptxdist gdb + (gdb) target remote 192.168.9.10:1234 + Remote debugging using 192.168.9.10:1234 + Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/bin/systemctl... + Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/lib/debug/.build-id/fb/3b770e637b51276dcdc41a1e8b37df8a6d6686.debug... + Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/lib/ld-linux-armhf.so.3... + Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/lib/debug/.build-id/0d/a8d3736dcd31fbf43ea0cb79d1f9699c6695ff.debug... + 0xb6f29624 in _start () from …/my-bsp/|ptxdistPlatformDir|/root/lib/ld-linux-armhf.so.3 + (gdb) break main + Breakpoint 1 at 0x4e3c90: file ../systemd-256.1/src/systemctl/systemctl.c, line 1372. + (gdb) continue + Continuing. + + Breakpoint 1, main (argc=1, argv=0xbe829d84) at ../systemd-256.1/src/systemctl/systemctl.c:1372 + 1372 DEFINE_MAIN_FUNCTION_WITH_POSITIVE_FAILURE(run); + (gdb) list + 1367 + 1368 /* Note that we return r here, not 0, so that we can implement the LSB-like return codes */ + 1369 return r; + 1370 } + 1371 + 1372 DEFINE_MAIN_FUNCTION_WITH_POSITIVE_FAILURE(run); + 1373 #endif + (gdb) + +You can now use GDB as usual, and when the debugged process terminates (or you +quit the GDB session), *gdbserver* will terminate as well, and close the +connection. + Migration between Releases -------------------------- -- 2.39.2