[ptxdist] ptxdist initial impressions

[Jookia <contact@jookia.org>]

Hello ptxdist friends,

I spent some time over the past few days trying to use ptxdist and
documenting all the pain I hit in the process. This isn't a criticism
more just anecdotal data of my experience. Hopefully it's useful to
smooth down some rough edges.

I'm going to write this up in a psuedo-markdown format in sections,
sorry if that breaks anyone's workflow.


Documentation
-------------

First up: Documentation is quite good aside from the 'Getting a working
environment' section. I will be focusing on this today since I think it's
probably the most critical problem with ptxdist at the moment: I expect
that by the end of following this chapter I should have a bootable image
of some kind and commands for using it, such as with qemu.

The first issue is that I'm told to downloaded files, but I'm not linked
to them, quoting:

ptxdist-2023.05.0.tar.bz2
OSELAS.BSP-Pengutronix-Example.tar.bz2 (or a similar source)
ptxdist-2019.09.0.tar.bz2
OSELAS.Toolchain-2019.09.1.tar.bz2

The OSELAS website is linked as ptxdist.org that doesn't mention OSELAS
at all. There is a link to the toolchain but it's a different version.
I managed to find a BSP for 'Generic' but it's 9 years old.

The documentation says having a BSP is optional, but there's no
instructions on how to work without one. It's probably not even a good
idea to mention this, perhaps ptxdist should provide a skeleton project?

I get the distinct impression the right solution here is to use
DistroKit as the starter BSP. This isn't mentioned anywhere in the
documentation but is something I found out about on IRC.

I'm also not a fan of suggesting installing ptxdist. It seems strange to
lock yourself to a repository of packages that you will need to tweak
and fix. I would like to see more support for running it in tree.

Likewise with installing a toolchain it seems a bit weird to suggest
Debian packages and going through how to extract these, but not offer a
tarball.

The page also talks about building a toolchain. This is kind of
a confusing process and seems way out of scope for getting up and
running. If things are this difficult maybe ptxdist should provide a
container or something with batteries included.

My setup
--------

That said I managed to get a setup going like this:

```
git clone https://git.pengutronix.de/git/ptxdist
git clone https://git.pengutronix.de/git/DistroKit
wget https://toolchains.bootlin.com/downloads/releases/toolchains/armv7-eabihf/tarballs/armv7-eabihf--glibc--bleeding-edge-2022.08-1.tar.bz2
tar -xvf armv7-eabihf--glibc--bleeding-edge-2022.08-1.tar.bz2
cd ptxdist
./autogen.sh
./configure
make -j8
cd ..
export PATH=$PATH:$PWD/ptxdist/bin
mkdir project
cd project
ln -s ../DistroKit base
ptxdist platform base/configs/platform-v7a/platformconfig
ptxdist toolchain ../armv7-eabihf--glibc--bleeding-edge-2022.08-1/bin
ptxdist migrate
ptxdist menu
# go to platformconfig -> architecture -> toolchain
# set toolchain to arm-buildroot-linux-gnueabihf
# disable toolchain version checks
# set glibc version to 2.35
rm selected_platformconfig
ln -s configs/platform-v7a/platformconfig selected_platformconfig
rm selected_toolchain/python* # bootlin weirdness?
```

This is pretty nasty, but worked for me and didn't require installing
anything.

I'm not sure why I needed to do some hack with selected_platformconfig.
I managed to get some strange error as my selected_platformconfig was
linked to the DistroKit configuration somehow.

I also had to be told on IRC that the Git link in DistroKit here:

https://www.pengutronix.de/en/software/distrokit.html

Is not meant to be cloned, it is a cgit repository. Cloning it is slow
without warning if you clone the cgit link.

If you don't select the correct glibc version and instead leave it empty
you will get errors during build.

It's also a little strange to me that migrate affected the DistroKit
repository. I guess it makes sense, but it means I've touched other
layers from my project. Having some kind of migration diff would be
nice.

Accessibility
-------------

The Kconfig used in ptxdist seems to have trouble with the speakup
screen reader. I haven't looked at how to fix this yet. It probably just
involves updating Kconfig. Worth mentioning through.

Build gripes
------------

Here's a quick list of gripes I had with the packages:

host-dtc failed to build with my toolchain, I had to edit the build and
remove -Werror from its code. Bumping dtc will probably fix this.

gdbserver gives this error:
ptxdist: error: md5sum for 'gdbserver' (/home/jookia/ptx/project/src/gdb-12.1.tar.xz) missing.
No clue how to fix this.

coreutils requires you selecting every package individually instead of
using them for your entire system. It also lacks egrep?

systemd didn't bring in IPTABLES_IPV4 so running it gets this error:
/usr/lib/systemd/systemd: error while loading shared libraries: libip4tc.so.2: cannot open shared object file: No such file or directory

Trying to build an image gives this error:
started : root.tgz
 * check_data_file_clashes: Package datapartition wants to install file /home/jookia/ptx/project/platform-v7a/build-target/image-root-tgz/usr/lib/udev/rules.d/91-mkfs-ext4.rules
        But that file is already provided by package  * systemd
 * opkg_solver_install: Cannot install package datapartition.

At this point I gave up trying to get a working image. It seems like
there's not much CI or testing done on running packages?

Testing roots
--------------

Testing images is important and I managed to use nfsroot and these
commands to test barebox and the kernel/rootfs:

qemu-system-arm -m 1024M -machine vexpress-a9 -cpu cortex-a9 -nographic -no-reboot -kernel platform-v7a/images/barebox-vexpress-ca9.img
qemu-system-arm -m 1024M -machine vexpress-a9 -cpu cortex-a9 -nographic -no-reboot -kernel platform-v7a/images/linuximage -dtb platform-v7a/images/vexpress-v2p-ca9.dtb -append 'console=ttyAMA0 ip=10.0.2.15::10.0.2.1:255.255.255.0 nfsroot=10.0.2.2:/root,v3,tcp,port=2049,mountport=2049 init=/bin/bash rw' -net nic,model=lan9118 -net user

These are much slower compared to testing an entire system. I tried
running this:

sudo systemd-nspawn --read-only --directory platform-v7a/root --boot

But systemd-nspawn failed to boot properly. I suspect this is because
permissions aren't mapped correctly. The proper way to fix this is
probably using user namespaces on the filesystem. This could be written
fairly easily I think.

I'm also unsure if it's a ptxdist issue or me issue, but ldd doesn't
work and says all the binaries are not dynamic executables.

Overall impressions
-------------------

Compared to buildroot's 'download and run make menuconfig' ptxdist has a
pretty hard hill to climb, especially with having to download toolchains
and BSPs and dealing with some of the package bugs.

I don't quite understand what's gained by having an installable ptxdist
and a DistroKit BSP if you're just going to modify both when building
your system. Maybe someone could explain this to me?

Thanks,
Jookia.