From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Thu, 19 Sep 2024 11:34:12 +0200 Received: from metis.whiteo.stw.pengutronix.de ([2a0a:edc0:2:b01:1d::104]) by lore.white.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1srDYC-005zgY-25 for lore@lore.pengutronix.de; Thu, 19 Sep 2024 11:34:12 +0200 Received: from localhost ([127.0.0.1] helo=metis.whiteo.stw.pengutronix.de) by metis.whiteo.stw.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1srDYB-0006YL-Op; Thu, 19 Sep 2024 11:34:11 +0200 Received: from drehscheibe.grey.stw.pengutronix.de ([2a0a:edc0:0:c01:1d::a2]) by metis.whiteo.stw.pengutronix.de with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1srDY2-0006Y6-Cp; Thu, 19 Sep 2024 11:34:02 +0200 Received: from [2a0a:edc0:0:1101:1d::54] (helo=dude05.red.stw.pengutronix.de) by drehscheibe.grey.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1srDY2-0091IJ-0J; Thu, 19 Sep 2024 11:34:02 +0200 Received: from mol by dude05.red.stw.pengutronix.de with local (Exim 4.96) (envelope-from ) id 1srDY1-0051tW-36; Thu, 19 Sep 2024 11:34:01 +0200 Date: Thu, 19 Sep 2024 11:34:01 +0200 From: Michael Olbrich To: Roland Hieber , ptxdist@pengutronix.de Message-ID: References: <20240823152620.954595-1-rhi@pengutronix.de> <20240826120418.2528432-1-rhi@pengutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20240826120418.2528432-1-rhi@pengutronix.de> X-Sent-From: Pengutronix Hildesheim X-URL: http://www.pengutronix.de/ X-IRC: #ptxdist @freenode X-Accept-Language: de,en X-Accept-Content-Type: text/plain Subject: Re: [ptxdist] [PATCH v2] doc: daily work: update debugging section X-BeenThere: ptxdist@pengutronix.de X-Mailman-Version: 2.1.29 Precedence: list List-Id: PTXdist Development Mailing List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ptxdist@pengutronix.de Sender: "ptxdist" X-SA-Exim-Connect-IP: 127.0.0.1 X-SA-Exim-Mail-From: ptxdist-bounces@pengutronix.de X-SA-Exim-Scanned: No (on metis.whiteo.stw.pengutronix.de); SAEximRunCond expanded to false On Mon, Aug 26, 2024 at 02:04:18PM +0200, Roland Hieber wrote: > * Split off the "Running QEMU" section into a stand-alone section > * Mention /usr/lib/debug instead of the old ./debug/ folders > * Describe `ptxdist gdb` and debugging on the target directly > * Keep GDB usage short; refer to third-party documentation > * Use literal blocks instead of code blocks without highlighting > * Make use of the |ptxdistPlatformDir| template variable > > Signed-off-by: Roland Hieber > --- > PATCH v2: > * somehow I forgot to stage & amend various changes before sending v1, > please disregard the old one > > > doc/daily_work.inc | 226 ++++++++++++++++++++++++++++++++++----------- > 1 file changed, 174 insertions(+), 52 deletions(-) > > diff --git a/doc/daily_work.inc b/doc/daily_work.inc > index b98adf75197d..65b0bad25d97 100644 > --- a/doc/daily_work.inc > +++ b/doc/daily_work.inc > @@ -285,15 +285,11 @@ nonexisting file, we can limit the output to all ``open`` system calls: > The binary may fail due to a missing ``/etc/foo.conf``. This could be a > hint on what is going wrong (it might not be the final solution). > > -Debugging with CPU emulation > ----------------------------- > - > -If we do not need some target related feature to run our application, we > -can also debug it through a simple CPU emulation. Thanks to QEMU we can > -run ELF binaries for other architectures than our build host is. > +Running an Application Made for a Different Architecture > +-------------------------------------------------------- > > -Running an Application made for a different Architecture > -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > +Thanks to QEMU we can run ELF binaries for different architectures than our > +build host. > > PTXdist creates a fully working root filesystem with all run-time > components in ``root/``. Lets assume we made a PTXdist based project for > @@ -318,64 +314,134 @@ host’s operating system. Using QEMU in this way let us simply check our > programs. There are also QEMU environments for other architectures > available. > > -Debugging an Application made for a different Architecture > -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > - > -Debugging our application is also possible with QEMU. All we need are a > -root filesystem with debug symbols available, QEMU and an architecture > -aware debugger. > - > -The root filesystem with debug symbols will be provided by PTXdist, the > -architecture aware debugger comes with the OSELAS.Toolchain. Two > -consoles are required for this debug session in this example. We start > -the QEMU in the first console as: > - > -.. code-block:: text > - > - $ cd ptxdistPlatformDir/root > - ptxdistPlatformDir/root$ qemu- -g 1234 -cpu -L . usr/bin/myapp > +Debugging an Application Made for a Different Architecture > +---------------------------------------------------------- > + > +Debugging an application is possible in three ways: > + > +#. :ref:`Directly running GDB on the target hardware `: > + This approach has the advantage that the target system is "self-contained", > + but it is necessary to install the debug symbols in the target's root file > + system, which can significantly improve the memory footprint. Furthermore, "improve" sounds wrong here. Maybe you meant "increase"? > + it is often not possible for GDB to access the application's source code > + unless further steps are taken (e.g. making the source code available in the > + correct paths over an NFS share or from a USB drive). > + > +#. :ref:`Running the application in a QEMU emulating the target architecture `: > + This approach doesn't require any additional hardware and can therefore be > + done completely on the build host, but it is limited to applications > + that do not depend on specific behaviour of the target hardware, like > + using external interfaces or architecture-specific features that are not > + available via QEMU (e.g. accessing GPS hardware over UART, or graphics > + acceleration). > + > +#. :ref:`Running gdbserver on the target hardware, and connecting to it with GDB > + from the build host `: > + This is the simplest use case when a target hardware with network interfaces > + is available, and PTXdist includes native support for it. > + In this case, the target file system only has to include the *gdbserver* > + binary, but does not need to contain any further debug information or source > + code, which are supplied by the build host. > + > +In any case, we need the respective binaries including their debug information, > +and a debugger for the target architecture. > +The root filesystem with debug symbols is provided by PTXdist in > +``|ptxdistPlatformDir|/root/`` on the build host, and the architecture-aware debugger > +comes with the OSELAS.Toolchain. > + > +.. note:: When PTXdist builds a target root file system, the binaries that are > + installed into it during the *targetinstall* stage of each package are > + always stripped of their debug symbols. Instead, the debug symbols are > + installed under ``|ptxdistPlatformDir|/root/usr/lib/debug/`` in separate files named > + after the *build ID* of the binary (which is a checksum over the contents of > + the binary, and can change when a binary is rebuilt). > + > + When GDB loads a binary, it will look for and automatically load matching > + debug symbols in ``usr/lib/debug`` relative to its *sysroot*, which is > + usually ``/``, but can be configured at runtime. > + > +.. note:: Information on using GDB is not part of this manual. > + There are countless cheatsheets available on the internet, but in most > + cases you will get around sufficiently with the ``break ``, > + ``run``, ``next``, ``step``, ``finish``, ``continue``, ``print ``, > + ``list `` and ``quit`` commands. > + Current GDB versions also include a well-structured online help with the > + ``help`` command, which can show you how to use each command. > + > +.. _gdb_on_target: > + > +Running GDB on the Target Hardware > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +In this case, GDB and the debug information need to be available on the target hardware. > +GDB can be enabled in the *menuconfig* with the ``PTXCONF_GDB`` symbol. > +Additionally, the debug symbols are installed into the target root file system > +when the ``PTXCONF_DEBUG_PACKAGES_INSTALL`` option is enabled. > +(If an image has been built before, i.e. the *targetinstall* stages have > +already been executed, a ``ptxdist clean root`` followed by a ``ptxdist go`` or > +``ptxdist images`` might be needed to install all debug symbols.) > + > +Due to the size of the resulting root file system, the target root file system > +is often supplied via :ref:`NFSROOT `. > + > +On the target, GDB can then be used normally (with the caveat that the source files are > +not available unless supplied manually):: > + > + root@target $ gdb /usr/bin/systemctl > + Reading symbols from /usr/bin/systemctl... > + Reading symbols from /usr/lib/debug/.build-id/fb/3b770e637b51276dcdc41a1e8b37df8a6d6686.debug... > + (gdb) break main > + Breakpoint 1 at 0x6c90: file ../systemd-256.1/src/systemctl/systemctl.c, line 1372. > + (gdb) run > + Starting program: /usr/bin/systemctl > + [Thread debugging using libthread_db enabled] > + Using host libthread_db library "/lib/libthread_db.so.1". > + > + Breakpoint 1, main (argc=1, argv=0xbee9bd64) > + at ../systemd-256.1/src/systemctl/systemctl.c:1372 > + 1372 in ../systemd-256.1/src/systemctl/systemctl.c > + (gdb) backtrace > + #0 main (argc=1, argv=0xbee9bd64) > + at ../systemd-256.1/src/systemctl/systemctl.c:1372 > + (gdb) list > + 1372 ../systemd-256.1/src/systemctl/systemctl.c: No such file or directory. > + (gdb) quit > + > +.. _gdb_via_qemu: > + > +Debugging with CPU Emulation > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +If we do not need target-specific features to run our application, we can also > +debug it through CPU emulation. > +Two consoles are required for this debug session in this example. We start > +the QEMU in the first console as:: > > -.. note:: PTXdist always builds a root filesystem ``root/``. > - It contains all components without debug > - information (all binaries are in the same size as used later on on the > - real target). In addition, each directory that contains binaries also > - contains a ``.debug/`` directory. It contains a file with only the debug > - symbols for each binary. These files are ignored while running > - applications but GDB knows about it and will automatically load the debug > - files. > + $ cd |ptxdistPlatformDir|/root > + |ptxdistPlatformDir|/root$ qemu- -g 1234 -cpu -L . usr/bin/myapp > > The added *-g 1234* parameter lets QEMU wait for a GDB connection to run > the application. > > In the second console we start GDB with the correct architecture > support. This GDB comes with the same OSELAS.Toolchain that was also > -used to build the project: > - > -.. code-block:: text > +used to build the project:: > > - $ ./selected_toolchain/-gdb --tui platform-/root/usr/bin/myapp > - > -This will run a *curses* based GDB. Not so easy to handle (we must enter > -all the commands and cannot click with a mouse!), but very fast to take > -a quick look at our application. > + $ ./selected_toolchain/-gdb |ptxdistPlatformDir|/root/usr/bin/myapp > > At first we tell GDB where to look for debug symbols. The correct > -directory here is ``root/``. > - > -.. code-block:: text > +directory here is ``|ptxdistPlatformDir|/root/``:: > > - (gdb) set solib-absolute-prefix platform-/root > + (gdb) set solib-absolute-prefix |ptxdistPlatformDir|/root This part of the documentation is pretty old and a bit outdated. Can you change it to use 'ptxdist gdb'? It should possible to use it in this case and then manually setting 'solib-absolute-prefix' is not needed. Michael > > -Next we connect this GDB to the waiting QEMU: > - > -.. code-block:: text > +Next we connect this GDB to the waiting QEMU:: > > (gdb) target remote localhost:1234 > Remote debugging using localhost:1234 > [New Thread 1] > 0x40096a7c in _start () from root/lib/ld.so.1 > > -As our application is already started, we can’t use the GDB command > +As our application is already started via QEMU, we can’t use the GDB command > ``start`` to run it until it reaches ``main()``. We set a breakpoint > instead at ``main()`` and *continue* the application: > > @@ -387,10 +453,6 @@ instead at ``main()`` and *continue* the application: > Continuing. > Breakpoint 1, main (argc=1, argv=0x4007f03c) at myapp.c:644 > > -The top part of the running gdbtui console will always show us the > -current source line. Due to the ``root/`` directory usage all > -debug information for GDB is available. > - > Now we can step through our application by using the commands *step*, > *next*, *stepi*, *nexti*, *until* and so on. > > @@ -403,6 +465,66 @@ Now we can step through our application by using the commands *step*, > debug symbols kept, it will be also possible for GDB to debug C library > functions our application calls (so it might worth the disk space). > > +.. _ptxdist_gdb: > + > +Remote-Debugging with *gdbserver* > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +When debugging with *gdbserver*, the debugging session is split into two parts: > +the server runs on the target and is remote-controlled by a GDB instance > +running on the build host over a network connection. > +You need to enable the ``PTXCONF_GDBSERVER`` symbol in PTXdists's *menuconfig* > +which makes *gdbserver* available on your target. > + > +Like for the approach above, you will need two terminals. > +In the first terminal you connect to your running target (e.g. via SSH or > +serial console), start *gdbserver* and tell it the binary to debug and where > +to listen for connections:: > + > + root@target $ gdbserver 192.168.1.11:1234 /usr/bin/systemctl > + Process /usr/bin/systemctl created; pid = 477 > + Listening on port 1234 > + > +In the above example, 192.168.1.11 is the outside-facing IP address of the > +target, and 1234 is the port on which to listen. > +gdbserver has now created a process, and waits for connections from a GDB. > + > +In the other terminal, you then start ``ptxdist gdb``, which will set up GDB > +correctly so that it can find all necessary debug symbols in PTXdist's > +``|ptxdistPlatformDir|/root/`` folder, and the sources in > +``|ptxdistPlatformDir|/sysroot-target/``. > +You then connect to the gdbserver instance running on the target using GDB's > +*target remote* command:: > + > + user@buildhost $ ptxdist gdb > + (gdb) target remote 192.168.9.10:1234 > + Remote debugging using 192.168.9.10:1234 > + Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/bin/systemctl... > + Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/lib/debug/.build-id/fb/3b770e637b51276dcdc41a1e8b37df8a6d6686.debug... > + Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/lib/ld-linux-armhf.so.3... > + Reading symbols from …/my-bsp/|ptxdistPlatformDir|/root/usr/lib/debug/.build-id/0d/a8d3736dcd31fbf43ea0cb79d1f9699c6695ff.debug... > + 0xb6f29624 in _start () from …/my-bsp/|ptxdistPlatformDir|/root/lib/ld-linux-armhf.so.3 > + (gdb) break main > + Breakpoint 1 at 0x4e3c90: file ../systemd-256.1/src/systemctl/systemctl.c, line 1372. > + (gdb) continue > + Continuing. > + > + Breakpoint 1, main (argc=1, argv=0xbe829d84) at ../systemd-256.1/src/systemctl/systemctl.c:1372 > + 1372 DEFINE_MAIN_FUNCTION_WITH_POSITIVE_FAILURE(run); > + (gdb) list > + 1367 > + 1368 /* Note that we return r here, not 0, so that we can implement the LSB-like return codes */ > + 1369 return r; > + 1370 } > + 1371 > + 1372 DEFINE_MAIN_FUNCTION_WITH_POSITIVE_FAILURE(run); > + 1373 #endif > + (gdb) > + > +You can now use GDB as usual, and when the debugged process terminates (or you > +quit the GDB session), *gdbserver* will terminate as well, and close the > +connection. > + > Migration between Releases > -------------------------- > > -- > 2.39.2 > > > -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |