Re: [ptxdist] ptxdist initial impressions

[Alexander Dahl <ada@thorsis.com>]

Hello Jookia,

as a long term user not affiliated with pengutronix I have some comments which 
might help.  See below.

Am Mittwoch, 17. Mai 2023, 18:11:31 CEST schrieb Jookia:
> Hello ptxdist friends,
> 
> I spent some time over the past few days trying to use ptxdist and
> documenting all the pain I hit in the process. This isn't a criticism
> more just anecdotal data of my experience. Hopefully it's useful to
> smooth down some rough edges.
> 
> I'm going to write this up in a psuedo-markdown format in sections,
> sorry if that breaks anyone's workflow.
> 
> 
> Documentation
> -------------
> 
> First up: Documentation is quite good aside from the 'Getting a working
> environment' section. I will be focusing on this today since I think it's
> probably the most critical problem with ptxdist at the moment: I expect
> that by the end of following this chapter I should have a bootable image
> of some kind and commands for using it, such as with qemu.
> 
> The first issue is that I'm told to downloaded files, but I'm not linked
> to them, quoting:
> 
> ptxdist-2023.05.0.tar.bz2
> OSELAS.BSP-Pengutronix-Example.tar.bz2 (or a similar source)
> ptxdist-2019.09.0.tar.bz2
> OSELAS.Toolchain-2019.09.1.tar.bz2
> 
> The OSELAS website is linked as ptxdist.org that doesn't mention OSELAS
> at all. There is a link to the toolchain but it's a different version.
> I managed to find a BSP for 'Generic' but it's 9 years old.

+1

That has to be improved.  There's no easy accessible single platform example 
BSP to get your feet wet, and DistroKit is nice for supporting some quite 
common SBCs, but is already quite complicated.

What's also lacking: some template or documentation on how to start from 
scratch without deriving your work from pre existing BSPs.  I tried that 
lately and put the steps online:

https://gist.github.com/LeSpocky/31af75ab63bc6f35fd71d53f06b5a50e

I dropped that link in IRC, but I got no clear signal if that's something to 
put into the official ptxdist documentation?

> The documentation says having a BSP is optional, but there's no
> instructions on how to work without one. It's probably not even a good
> idea to mention this, perhaps ptxdist should provide a skeleton project?
> 
> I get the distinct impression the right solution here is to use
> DistroKit as the starter BSP. This isn't mentioned anywhere in the
> documentation but is something I found out about on IRC.
> 
> I'm also not a fan of suggesting installing ptxdist. It seems strange to
> lock yourself to a repository of packages that you will need to tweak
> and fix. I would like to see more support for running it in tree.

Maybe this is some kind of misunderstanding.  You would never touch the 
distributed/installed ptxdist, but "overwrite" things by placing package rules 
with the same name in your BSP.  For example if you need a version bump for 
libfoo, you could copy /usr/local/lib/ptxdist-2100.01.0/rules/libfoo.make to 
your BSP at rules/libfoo.make and adapt it to your needs.

> Likewise with installing a toolchain it seems a bit weird to suggest
> Debian packages and going through how to extract these, but not offer a
> tarball.
> 
> The page also talks about building a toolchain. This is kind of
> a confusing process and seems way out of scope for getting up and
> running. If things are this difficult maybe ptxdist should provide a
> container or something with batteries included.
> 
> My setup
> --------
> 
> That said I managed to get a setup going like this:
> 
> ```
> git clone https://git.pengutronix.de/git/ptxdist
> git clone https://git.pengutronix.de/git/DistroKit
> wget
> https://toolchains.bootlin.com/downloads/releases/toolchains/armv7-eabihf/t
> arballs/armv7-eabihf--glibc--bleeding-edge-2022.08-1.tar.bz2 tar -xvf
> armv7-eabihf--glibc--bleeding-edge-2022.08-1.tar.bz2
> cd ptxdist
> ./autogen.sh
> ./configure
> make -j8
> cd ..
> export PATH=$PATH:$PWD/ptxdist/bin
> mkdir project
> cd project
> ln -s ../DistroKit base
> ptxdist platform base/configs/platform-v7a/platformconfig
> ptxdist toolchain ../armv7-eabihf--glibc--bleeding-edge-2022.08-1/bin
> ptxdist migrate
> ptxdist menu
> # go to platformconfig -> architecture -> toolchain
> # set toolchain to arm-buildroot-linux-gnueabihf
> # disable toolchain version checks
> # set glibc version to 2.35
> rm selected_platformconfig
> ln -s configs/platform-v7a/platformconfig selected_platformconfig

I think this is a misunderstanding from using layers.  First you select base/
configs/platform-v7a/platformconfig as your platformconfig.  Then with ptxdist 
menu -> platformconfig you make changes, but those are not put to the base 
layer, but to your top layer.  ptxdist creates configs/platform-v7a/
platformconfig and configs/platform-v7a/platformconfig.diff then.  On the next 
invocation ptxdist detects your selected_platformconfig points to the base 
layer although you have a different platformconfig in the top layer, and it 
probably reports that in some way.  You could have just done this:

  ptxdist platform configs/platform-v7a/platformconfig

But I would agree this is a rough edge.  Not sure if it can be solved with 
code or with documentation?

> rm selected_toolchain/python* # bootlin weirdness?
> ```
> 
> This is pretty nasty, but worked for me and didn't require installing
> anything.
> 
> I'm not sure why I needed to do some hack with selected_platformconfig.
> I managed to get some strange error as my selected_platformconfig was
> linked to the DistroKit configuration somehow.

Well, see above.

> 
> I also had to be told on IRC that the Git link in DistroKit here:
> 
> https://www.pengutronix.de/en/software/distrokit.html
> 
> Is not meant to be cloned, it is a cgit repository. Cloning it is slow
> without warning if you clone the cgit link.
> 
> If you don't select the correct glibc version and instead leave it empty
> you will get errors during build.
> 
> It's also a little strange to me that migrate affected the DistroKit
> repository. I guess it makes sense, but it means I've touched other
> layers from my project. Having some kind of migration diff would be
> nice.

It should not affect the base layer.  Strange.

> Accessibility
> -------------
> 
> The Kconfig used in ptxdist seems to have trouble with the speakup
> screen reader. I haven't looked at how to fix this yet. It probably just
> involves updating Kconfig. Worth mentioning through.
> 
> Build gripes
> ------------
> 
> Here's a quick list of gripes I had with the packages:
> 
> host-dtc failed to build with my toolchain, I had to edit the build and
> remove -Werror from its code. Bumping dtc will probably fix this.
> 
> gdbserver gives this error:
> ptxdist: error: md5sum for 'gdbserver'
> (/home/jookia/ptx/project/src/gdb-12.1.tar.xz) missing. No clue how to fix
> this.

gdbserver is special.  Usually gdb is part of the toolchain.  ptxdist does 
some magic here.  Go to menuconfig -> Debug Tools -> gdb and on top you see to 
Kconfig entries which should help you: "use toolchain gdb version" and "gdb 
source md5sum".  

> coreutils requires you selecting every package individually instead of
> using them for your entire system. It also lacks egrep?
> 
> systemd didn't bring in IPTABLES_IPV4 so running it gets this error:
> /usr/lib/systemd/systemd: error while loading shared libraries:
> libip4tc.so.2: cannot open shared object file: No such file or directory
> 
> Trying to build an image gives this error:
> started : root.tgz
>  * check_data_file_clashes: Package datapartition wants to install file
> /home/jookia/ptx/project/platform-v7a/build-target/image-root-tgz/usr/lib/u
> dev/rules.d/91-mkfs-ext4.rules But that file is already provided by package 
> * systemd
>  * opkg_solver_install: Cannot install package datapartition.
> 
> At this point I gave up trying to get a working image. It seems like
> there's not much CI or testing done on running packages?

I'm not aware of any public service like that.

> Testing roots
> --------------
> 
> Testing images is important and I managed to use nfsroot and these
> commands to test barebox and the kernel/rootfs:
> 
> qemu-system-arm -m 1024M -machine vexpress-a9 -cpu cortex-a9 -nographic
> -no-reboot -kernel platform-v7a/images/barebox-vexpress-ca9.img
> qemu-system-arm -m 1024M -machine vexpress-a9 -cpu cortex-a9 -nographic
> -no-reboot -kernel platform-v7a/images/linuximage -dtb
> platform-v7a/images/vexpress-v2p-ca9.dtb -append 'console=ttyAMA0
> ip=10.0.2.15::10.0.2.1:255.255.255.0
> nfsroot=10.0.2.2:/root,v3,tcp,port=2049,mountport=2049 init=/bin/bash rw'
> -net nic,model=lan9118 -net user
> 
> These are much slower compared to testing an entire system. I tried
> running this:
> 
> sudo systemd-nspawn --read-only --directory platform-v7a/root --boot
> 
> But systemd-nspawn failed to boot properly. I suspect this is because
> permissions aren't mapped correctly. The proper way to fix this is
> probably using user namespaces on the filesystem. This could be written
> fairly easily I think.
> 
> I'm also unsure if it's a ptxdist issue or me issue, but ldd doesn't
> work and says all the binaries are not dynamic executables.
> 
> Overall impressions
> -------------------
> 
> Compared to buildroot's 'download and run make menuconfig' ptxdist has a
> pretty hard hill to climb, especially with having to download toolchains
> and BSPs and dealing with some of the package bugs.
> 
> I don't quite understand what's gained by having an installable ptxdist
> and a DistroKit BSP if you're just going to modify both when building
> your system. Maybe someone could explain this to me?

You don't need to modify those.  I explained it for ptxdist itself already.  
For DistroKit it's easier to just use it as a base layer as is, and not use 
DistroKit directly as your BSP.

Most problems I see are with documentation.  Question is: who reviews the 
existing one and writes the missing pieces?  Patches for documentation can be 
sent to the mailing list as every other patch.

HTH & Greets
Alex