Re: [ptxdist] Segmentation fault with ptxdist-2020.05.0

[Mircea Ciocan <m.ciocan@ppc-ag.de>]

On 15.05.20 15:02, Mircea Ciocan wrote:
>
> On 15.05.20 09:46, Michael Olbrich wrote:
>> On Thu, May 14, 2020 at 03:36:44PM +0200, Mircea Ciocan wrote:
>>> Hello list,
>>>
>>> I have the most strange error with the ptxdist-2020.05.0 while 
>>> trying to
>>> compile a simple image for the iMX6ULL EVK, the shell segfaults !!!
>>>
>>> Here are some information, that you could find useful:
>>>
>>> Distribution: Ubuntu 18.04 LTS, 8GB RAM, current updates.
>>>
>>> System shell set to bash.
>>>
>>> The compilation of everything goes perfectly OK, but while building 
>>> the root
>>> filesystem image it crashes, always in the same place (100% 
>>> reproducible),
>>> when adding the net-dns-perl package, with the following message;
>>>
>>> info: selected ptxconfig:
>>>        'configs/platform-mcimx6ull-evk/ptxconfig.tiny'
>>> ptxdist: warning: No dev packages found in 
>>> 'Firmware/packages-MCIMX6ULL-EVK'
>>>
>>> ...[snip]...
>>>
>>> [compilation goes OK]
>>>
>>> ----------------
>>> target: root.tgz
>>> ----------------
>>> ...[snip]...
>>>
>>> Configuring readline.
>>> Configuring net-dns-perl.
>>> environment: line 1: 31523 Done                    { echo "cd 
>>> '${work_dir}'
>>> || exit"; ptxd_dopermissions "${ptxd_reply_perm_files[@]}"; echo ":"; }
>>>       31524 Segmentation fault      (core dumped) | sh
>>> ./Software/ptxdist-2020.05.0/rules/image-root-tgz.make:29: recipe 
>>> for target
>>> './Software/Firmware/platform-MCIMX6ULL-EVK/images/root.tgz' failed
>>>
>>> make: *** 
>>> [./Software/Firmware/platform-MCIMX6ULL-EVK/images/root.tgz] Error
>>> 139
>>>
>>>
>>> In the system log I get something like:
>>>
>>> "vmunix: [132753.581596] sh[10747]: segfault at 7ffe5bdf1fe8 ip
>>> 00007fdb0b04930e sp 00007ffe5bdf1fe0 error 6 in
>>> libc-2.27.so[7fdb0afb5000+1e7000]"
>>>
>>>
>>> I have to say that I rarely, if ever, see this type of crash, and 
>>> the build
>>> finishes OK with ptxdist-2020.02.0. Any pointer on how can I fix or 
>>> debug
>>> this are greatly appreciated.
>> Huh, I've never seen anything like this. Some things to test.
>>
>> Take a look at scripts/lib/ptxd_make_image_prepare_work_dir.sh in 
>> PTXdist.
>> The relevant code is there. Line 50ff.
>> If you change line 56 to something like this:
>>
>> } | tee /tmp/ptxdist-sh-test.txt | sh &&
>>
>> Then this will save the exact code that the shell executes when it 
>> crashed.
>> Maybe we can find some clues there. But It's probably a bug in 'sh'.
>>
>> What's your /bin/sh anyways? For me it's a link to 'dash' but sometimes
>> it's a bash. You could explicitly try the other here instead of 'sh' 
>> to see
>> if that crashes as well.
>>
>> Michael
>>
> Hello Michael and thanks for the hint, first things first, the shell 
> is the GNU bash, I've change it via the update-alternatives, because 
> more than one time I've had issues with scripts that call /bin/sh, but 
> assume that is bash.
>
> The logging tap collected the following file:
>
> "cd 
> './Software/Firmware/platform-MCIMX6ULL-EVK/build-target/image-root-tgz' 
> || exit
> chmod 0755    './' &&
> chown 0.0    './' &&
> chmod 0755    './usr' &&
> chown 0.0    './usr' &&
> chmod 0755    './usr/sbin' &&
> chown 0.0    './usr/sbin' &&
>
> ...
>
> !!! Another >>>71940<<< similar lines !!!
>
> ...
>
> chmod 0755    './usr/bin/cpuburn-a9' &&
> chown 0.0    './usr/bin/cpuburn-a9' &&
> :"
>
> Is this just a "cool way" to execute a quatrillion of commands, by 
> exploiting to the limit the AND_IF (&&) construct to exit on error, or 
> there is a deeper purpose ?
>
> I could see that some other cool programmer tried some optimization 
> somewhere in bash and these two worlds collided hard, now that full 
> Perl and Python packages adds a few tens of thousands of files to the 
> image that need their permissions fixed.
>
> I'll have a look if there was another way to do it (how was it done in 
> 2020.03.0 and lower, could it be that only two lines have been used 
> like: chmod -R * && chown -R 0.0 * und fertig ;) ?!?) and see if I 
> could reproduce it, to make an Ubuntu bug report.
>
> Other ideas and solutions are most welcome.
>
>
> Thanks for your help,
>
> Mircea 

OK, latest update, is definitly a (Ubuntu) bash bug, I've run the 
mega-list of commands with bash and dash, bash crashes immediately, dash 
takes a mounstrously large amount of time (almost 5min on a beefy 
machine with fast SSD), so it is clear why Michael did't see the bug.
A bit of stracing show something like a lot of threads dieing, so 
probably bash tries to execute this in parallel and hits something, just 
FYI here the last words of bash run strace:

read(255, "mod 0755\t'./usr' &&\nchown 0.0\t'."..., 8192) = 8192
brk(0x56519329b000)                     = 0x56519329b000
read(255, "tch_map' &&\nchmod 0755\t'./' &&\nc"..., 8192) = 2374
brk(0x5651932bc000)                     = 0x5651932bc000
brk(0x5651932dd000)                     = 0x5651932dd000
.... maaaany more similar lines, with different addresses ....
brk(0x565193805000)                     = 0x565193805000
brk(0x565193826000)                     = 0x565193826000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, 
si_addr=0x7ffef981df98} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)

So because switching back to dash here it's a bit difficult, is there 
any way that this function could be made a bit more bash-friendly ?
I could gladly test your patches and ideas (vanilla Ubuntu 18.04, with 
bash as system shell, is not such a seldom encountered configuration), 
but I'm afraid that I'm not in the deep of the PTXdist internals to 
start hacking and don't produce conequences.

  Best regards,
  Mircea




_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de