Re: [ptxdist] Segmentation fault with ptxdist-2020.05.0

[Michael Olbrich <m.olbrich@pengutronix.de>]

On Fri, May 15, 2020 at 03:36:30PM +0200, Mircea Ciocan wrote:
> 
> On 15.05.20 15:02, Mircea Ciocan wrote:
> > 
> > On 15.05.20 09:46, Michael Olbrich wrote:
> > > On Thu, May 14, 2020 at 03:36:44PM +0200, Mircea Ciocan wrote:
> > > > Hello list,
> > > > 
> > > > I have the most strange error with the ptxdist-2020.05.0 while
> > > > trying to
> > > > compile a simple image for the iMX6ULL EVK, the shell segfaults !!!
> > > > 
> > > > Here are some information, that you could find useful:
> > > > 
> > > > Distribution: Ubuntu 18.04 LTS, 8GB RAM, current updates.
> > > > 
> > > > System shell set to bash.
> > > > 
> > > > The compilation of everything goes perfectly OK, but while
> > > > building the root
> > > > filesystem image it crashes, always in the same place (100%
> > > > reproducible),
> > > > when adding the net-dns-perl package, with the following message;
> > > > 
> > > > info: selected ptxconfig:
> > > >        'configs/platform-mcimx6ull-evk/ptxconfig.tiny'
> > > > ptxdist: warning: No dev packages found in
> > > > 'Firmware/packages-MCIMX6ULL-EVK'
> > > > 
> > > > ...[snip]...
> > > > 
> > > > [compilation goes OK]
> > > > 
> > > > ----------------
> > > > target: root.tgz
> > > > ----------------
> > > > ...[snip]...
> > > > 
> > > > Configuring readline.
> > > > Configuring net-dns-perl.
> > > > environment: line 1: 31523 Done                    { echo "cd
> > > > '${work_dir}'
> > > > || exit"; ptxd_dopermissions "${ptxd_reply_perm_files[@]}"; echo ":"; }
> > > >       31524 Segmentation fault      (core dumped) | sh
> > > > ./Software/ptxdist-2020.05.0/rules/image-root-tgz.make:29:
> > > > recipe for target
> > > > './Software/Firmware/platform-MCIMX6ULL-EVK/images/root.tgz' failed
> > > > 
> > > > make: ***
> > > > [./Software/Firmware/platform-MCIMX6ULL-EVK/images/root.tgz]
> > > > Error
> > > > 139
> > > > 
> > > > 
> > > > In the system log I get something like:
> > > > 
> > > > "vmunix: [132753.581596] sh[10747]: segfault at 7ffe5bdf1fe8 ip
> > > > 00007fdb0b04930e sp 00007ffe5bdf1fe0 error 6 in
> > > > libc-2.27.so[7fdb0afb5000+1e7000]"
> > > > 
> > > > 
> > > > I have to say that I rarely, if ever, see this type of crash,
> > > > and the build
> > > > finishes OK with ptxdist-2020.02.0. Any pointer on how can I fix
> > > > or debug
> > > > this are greatly appreciated.
> > > Huh, I've never seen anything like this. Some things to test.
> > > 
> > > Take a look at scripts/lib/ptxd_make_image_prepare_work_dir.sh in
> > > PTXdist.
> > > The relevant code is there. Line 50ff.
> > > If you change line 56 to something like this:
> > > 
> > > } | tee /tmp/ptxdist-sh-test.txt | sh &&
> > > 
> > > Then this will save the exact code that the shell executes when it
> > > crashed.
> > > Maybe we can find some clues there. But It's probably a bug in 'sh'.
> > > 
> > > What's your /bin/sh anyways? For me it's a link to 'dash' but sometimes
> > > it's a bash. You could explicitly try the other here instead of 'sh'
> > > to see
> > > if that crashes as well.
> > > 
> > > Michael
> > > 
> > Hello Michael and thanks for the hint, first things first, the shell is
> > the GNU bash, I've change it via the update-alternatives, because more
> > than one time I've had issues with scripts that call /bin/sh, but assume
> > that is bash.
> > 
> > The logging tap collected the following file:
> > 
> > "cd
> > './Software/Firmware/platform-MCIMX6ULL-EVK/build-target/image-root-tgz'
> > || exit
> > chmod 0755    './' &&
> > chown 0.0    './' &&
> > chmod 0755    './usr' &&
> > chown 0.0    './usr' &&
> > chmod 0755    './usr/sbin' &&
> > chown 0.0    './usr/sbin' &&
> > 
> > ...
> > 
> > !!! Another >>>71940<<< similar lines !!!
> > 
> > ...
> > 
> > chmod 0755    './usr/bin/cpuburn-a9' &&
> > chown 0.0    './usr/bin/cpuburn-a9' &&
> > :"
> > 
> > Is this just a "cool way" to execute a quatrillion of commands, by
> > exploiting to the limit the AND_IF (&&) construct to exit on error, or
> > there is a deeper purpose ?

It's just there to catch error. There shouldn't be any, but we don't want
to ignore the either.

> > I could see that some other cool programmer tried some optimization
> > somewhere in bash and these two worlds collided hard, now that full Perl
> > and Python packages adds a few tens of thousands of files to the image
> > that need their permissions fixed.
> > 
> > I'll have a look if there was another way to do it (how was it done in
> > 2020.03.0 and lower, could it be that only two lines have been used
> > like: chmod -R * && chown -R 0.0 * und fertig ;) ?!?) and see if I could

No, not all files have the same ownership/permissions. We save those from
the 'instal_*' commands in targetinstall and reapply here.
In this case, not everything is actually needed. So maybe we can shorten
the list.

> > reproduce it, to make an Ubuntu bug report.
> > 
> > Other ideas and solutions are most welcome.
> > 
> > 
> > Thanks for your help,
> > 
> > Mircea
> 
> OK, latest update, is definitly a (Ubuntu) bash bug, I've run the mega-list
> of commands with bash and dash, bash crashes immediately, dash takes a
> mounstrously large amount of time (almost 5min on a beefy machine with fast
> SSD), so it is clear why Michael did't see the bug.
> A bit of stracing show something like a lot of threads dieing, so probably
> bash tries to execute this in parallel and hits something, just FYI here the
> last words of bash run strace:
> 
> read(255, "mod 0755\t'./usr' &&\nchown 0.0\t'."..., 8192) = 8192
> brk(0x56519329b000)                     = 0x56519329b000
> read(255, "tch_map' &&\nchmod 0755\t'./' &&\nc"..., 8192) = 2374
> brk(0x5651932bc000)                     = 0x5651932bc000
> brk(0x5651932dd000)                     = 0x5651932dd000
> .... maaaany more similar lines, with different addresses ....
> brk(0x565193805000)                     = 0x565193805000
> brk(0x565193826000)                     = 0x565193826000
> --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x7ffef981df98}
> ---
> +++ killed by SIGSEGV (core dumped) +++
> Segmentation fault (core dumped)
> 
> So because switching back to dash here it's a bit difficult, is there any
> way that this function could be made a bit more bash-friendly ?
> I could gladly test your patches and ideas (vanilla Ubuntu 18.04, with bash
> as system shell, is not such a seldom encountered configuration), but I'm
> afraid that I'm not in the deep of the PTXdist internals to start hacking
> and don't produce conequences.

Hmmm, in this case, we can probably use 'set -e' instead. But I probably
want to avoid doing most of this stuff here in the first place. I need to
think about this.

Michael

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de