[ptxdist] weird ptxdist -q -j go problems

[ptxdist] weird ptxdist -q -j go problems

[Erwin Rol]

Hey all,

how wel is the -j option of ptxdist tested ? Is it used a lot by others
or is it more an obscure untested option ?

With the lastest version of ptxdist I wanted to try the -j option and
am running into weird problems. 

First I forgot to add a _CONF_TOOL setting in one of my *.make files,
and I found a file called "11" in my dir with the following content;

ptxdist: error: automatic prepare tool selection failed. Set <PKG>_CONF_TOOL

that "11" somehow sounds like it should have been a filedescriptor
number and not an actual file. 


Of course not having the _CONF_TOOL was an error from my side, after
fixing it it seemed to work.

But now the ptxdist -j -q go from time to time stops with an error. In
the log file it always seems to say something like;

}}} 2018-02-25T08:38:46+0000
/home/ANNAX.LOCAL/erro/Projects/ABE.centos7/bsps/Seco_E38XX_64/master/p
txdist/bin/ptxdist go -j -q; (exit value: 2)


I have seen this with busybox and openssl with different compilers
(x86_62 and armv5). It always is exit value: 2.

When I than do a ./ptdixt go again it happily continues (even if I use
-q -j) 

Any idea what is causing this? 

I hat this under Centos7 with a 24core and 32core machine. 

- Erwin


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] weird ptxdist -q -j go problems

[Michael Olbrich]

Hi,

On Sun, Feb 25, 2018 at 10:17:46AM +0100, Erwin Rol wrote:
> how wel is the -j option of ptxdist tested ? Is it used a lot by others
> or is it more an obscure untested option ?

I always use '-j -q', unless I test specific packages or need to debug
build error.

> With the lastest version of ptxdist I wanted to try the -j option and
> am running into weird problems. 
> 
> First I forgot to add a _CONF_TOOL setting in one of my *.make files,
> and I found a file called "11" in my dir with the following content;
> 
> ptxdist: error: automatic prepare tool selection failed. Set <PKG>_CONF_TOOL
> 
> that "11" somehow sounds like it should have been a filedescriptor
> number and not an actual file. 

I think this is an issue with your bash version. The code for this message
is this:
[...]
echo "${PTXDIST_LOG_PROMPT}error: $1" >&${PTXDIST_FD_STDERR}
[...]

I think older bash versions have a problem with this under some
circumstances. I don't remember all the details, but it may be triggered by
the call chain 'bash -> make -> bash'. The fd is  opened in the first bash
and used in the second.

> Of course not having the _CONF_TOOL was an error from my side, after
> fixing it it seemed to work.
> 
> But now the ptxdist -j -q go from time to time stops with an error. In
> the log file it always seems to say something like;
> 
> }}} 2018-02-25T08:38:46+0000
> /home/ANNAX.LOCAL/erro/Projects/ABE.centos7/bsps/Seco_E38XX_64/master/p
> txdist/bin/ptxdist go -j -q; (exit value: 2)
> 
> 
> I have seen this with busybox and openssl with different compilers
> (x86_62 and armv5). It always is exit value: 2.
> 
> When I than do a ./ptdixt go again it happily continues (even if I use
> -q -j) 

Can you send the relevant parts from the logfile? I know there is a
dependency problem in openssl. I cause very rare build errors. I've yet to
figure out the exact cause. It's not '-j' specific but that seems to
trigger it more often.

I'm not aware of any problems with busybox. A log with some more details
would be helpful.

> Any idea what is causing this? 
> 
> I hat this under Centos7 with a 24core and 32core machine. 

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] weird ptxdist -q -j go problems

[Erwin Rol]

Hey Michael,

On Sun, 2018-02-25 at 11:38 +0100, Michael Olbrich wrote:
> Hi,
> 
> On Sun, Feb 25, 2018 at 10:17:46AM +0100, Erwin Rol wrote:
> > how wel is the -j option of ptxdist tested ? Is it used a lot by others
> > or is it more an obscure untested option ?
> 
> I always use '-j -q', unless I test specific packages or need to debug
> build error.
> 

OK that is good :-) 

> > With the lastest version of ptxdist I wanted to try the -j option and
> > am running into weird problems. 
> > 
> > First I forgot to add a _CONF_TOOL setting in one of my *.make files,
> > and I found a file called "11" in my dir with the following content;
> > 
> > ptxdist: error: automatic prepare tool selection failed. Set <PKG>_CONF_TOOL
> > 
> > that "11" somehow sounds like it should have been a filedescriptor
> > number and not an actual file. 
> 
> I think this is an issue with your bash version. The code for this message
> is this:
> [...]
> echo "${PTXDIST_LOG_PROMPT}error: $1" >&${PTXDIST_FD_STDERR}
> [...]
> 
> I think older bash versions have a problem with this under some
> circumstances. I don't remember all the details, but it may be triggered by
> the call chain 'bash -> make -> bash'. The fd is  opened in the first bash
> and used in the second.

The build system is Centos7 with bash;
GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)

Not the newest but on the other hand Centos7/RHEL7 is a very stable
platform. It would be very nice if it could be part of the list of
platforms Pengutronix uses to test ptxdist ;-)

> 
> > Of course not having the _CONF_TOOL was an error from my side, after
> > fixing it it seemed to work.
> > 
> > But now the ptxdist -j -q go from time to time stops with an error. In
> > the log file it always seems to say something like;
> > 
> > }}} 2018-02-25T08:38:46+0000
> > /home/ANNAX.LOCAL/erro/Projects/ABE.centos7/bsps/Seco_E38XX_64/master/p
> > txdist/bin/ptxdist go -j -q; (exit value: 2)
> > 
> > 
> > I have seen this with busybox and openssl with different compilers
> > (x86_62 and armv5). It always is exit value: 2.
> > 
> > When I than do a ./ptdixt go again it happily continues (even if I use
> > -q -j) 
> 
> Can you send the relevant parts from the logfile? I know there is a
> dependency problem in openssl. I cause very rare build errors. I've yet to
> figure out the exact cause. It's not '-j' specific but that seems to
> trigger it more often.
> 
> I'm not aware of any problems with busybox. A log with some more details
> would be helpful.

I found a missing dependency for libsoup, it needs glib host tools. I
am running from a minimal docker so those aren't available on the host
and so libsoup.in needed select HOST_GLIB

Maybe it is just caused by some other missing deps. 

BTW: wouldn't it be easier to add select HOST_GLIB to glib.in ? Or even
more general do that for all packages that have a host-* version ?

I'll look into it and report back when I know more, certainly now I
know -j really is a first class citizen :-)

- Erwin


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] weird ptxdist -q -j go problems

[Michael Olbrich]

Hi,

On Sun, Feb 25, 2018 at 11:56:47AM +0100, Erwin Rol wrote:
> The build system is Centos7 with bash;
> GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
> 
> Not the newest but on the other hand Centos7/RHEL7 is a very stable
> platform. It would be very nice if it could be part of the list of
> platforms Pengutronix uses to test ptxdist ;-)

I run build tests in several different chroots and Centos7 is one of them.
Unfortunately that one is currently broken, because it's missing Python 3.4
needed for meson :-/.

> > > Of course not having the _CONF_TOOL was an error from my side, after
> > > fixing it it seemed to work.
> > > 
> > > But now the ptxdist -j -q go from time to time stops with an error. In
> > > the log file it always seems to say something like;
> > > 
> > > }}} 2018-02-25T08:38:46+0000
> > > /home/ANNAX.LOCAL/erro/Projects/ABE.centos7/bsps/Seco_E38XX_64/master/p
> > > txdist/bin/ptxdist go -j -q; (exit value: 2)
> > > 
> > > 
> > > I have seen this with busybox and openssl with different compilers
> > > (x86_62 and armv5). It always is exit value: 2.
> > > 
> > > When I than do a ./ptdixt go again it happily continues (even if I use
> > > -q -j) 
> > 
> > Can you send the relevant parts from the logfile? I know there is a
> > dependency problem in openssl. I cause very rare build errors. I've yet to
> > figure out the exact cause. It's not '-j' specific but that seems to
> > trigger it more often.
> > 
> > I'm not aware of any problems with busybox. A log with some more details
> > would be helpful.
> 
> I found a missing dependency for libsoup, it needs glib host tools. I
> am running from a minimal docker so those aren't available on the host
> and so libsoup.in needed select HOST_GLIB
> 
> Maybe it is just caused by some other missing deps. 

Some packages using glib also need host-glib for some of the helper tools
like glib-mkenums.

Can you send a patch? And please include the reason in the commit message.
Or the build error if it's useful. That helps while reviewing the
dependencies for the next update.

> BTW: wouldn't it be easier to add select HOST_GLIB to glib.in ? Or even
> more general do that for all packages that have a host-* version ?

Not all packages that use glib actually need host-glib and I'd like to
avoid unnecessary dependenciesa.

> I'll look into it and report back when I know more, certainly now I
> know -j really is a first class citizen :-)

I've spent quite some time to make '-j' work without problems and as fast
as possible without overloading the build server. All my ptxdist build
tests use it. I's most definitely a first class citizen.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] weird ptxdist -q -j go problems

[Erwin Rol]

On Sun, 2018-02-25 at 12:43 +0100, Michael Olbrich wrote:
> Hi,
> 
> On Sun, Feb 25, 2018 at 11:56:47AM +0100, Erwin Rol wrote:
> > The build system is Centos7 with bash;
> > GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
> > 
> > Not the newest but on the other hand Centos7/RHEL7 is a very stable
> > platform. It would be very nice if it could be part of the list of
> > platforms Pengutronix uses to test ptxdist ;-)
> 
> I run build tests in several different chroots and Centos7 is one of them.
> Unfortunately that one is currently broken, because it's missing Python 3.4
> needed for meson :-/.

install devtoolset-6, this is what I install in my centos7 docker;

RUN \
    echo 'Upgrading all system packages to the latest available versions' >&2 && \
    yum -y install epel-release && \
    yum -y install centos-release-scl && \
    yum -y install devtoolset-6 && \
    yum -y update
    
RUN \
    echo 'Installing host tools and build system functionality' >&2 && \
    yum -y install \
        cmake curl cppcheck \
        gawk make wget tar bzip2 gzip unzip perl patch \
        diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath socat \
        perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue xz which \
        ncurses-devel flex bison texinfo gettext expect dialog gperf imake \
        xmlto-tex libffi-devel ruby autogen automake bc lzop sudo \
        perl-XML-Parser intltool parted tig htop vim file \
        perl-XML-LibXML perl-JSON \
        libX11-devel openssl-devel glibc.i686 libffi-devel \
        perl-Carp perl-Encode perl-Exporter perl-File-Path perl-File-Temp perl-Filter \
        perl-Getopt-Long perl-HTTP-Tiny perl-PathTools perl-Pod-Escapes perl-Pod-Perldoc \
        perl-Pod-Simple perl-Pod-Usage perl-Scalar-List-Utils perl-Socket perl-Storable \
        perl-Text-ParseWords perl-Time-Local perl-constant perl-libs perl-macros perl-parent \
        perl-podlators perl-threads perl-threads-shared runtime perl-Data-Dumper \
        perl-Thread-Queue perl-JSON-PP emacs unix2dos dos2unix \
        dosfstools mtools pxz gtk-doc elfutils-libelf-devel subversion git-svn \
        python34 python34-devel python34-pip python34-libs \
        python python-mako python-devel libxml2-python python-argparse

> Some packages using glib also need host-glib for some of the helper tools
> like glib-mkenums.
> 
> Can you send a patch? And please include the reason in the commit message.
> Or the build error if it's useful. That helps while reviewing the
> dependencies for the next update.

I will have to break it again to get the error msg :-) I'll see if I
find some time later today.

> > BTW: wouldn't it be easier to add select HOST_GLIB to glib.in ? Or even
> > more general do that for all packages that have a host-* version ?
> 
> Not all packages that use glib actually need host-glib and I'd like to
> avoid unnecessary dependenciesa.

I'll see if I can find some time to do  
./ptxdist clean; ./ptxdist compile pkg_that_uses_glib 
to see what builds and waht doesn't.

Of course that is going to cost a lot of build time, so it might take a
while to get done.

> > I'll look into it and report back when I know more, certainly now I
> > know -j really is a first class citizen :-)
> 
> I've spent quite some time to make '-j' work without problems and as fast
> as possible without overloading the build server. All my ptxdist build
> tests use it. I's most definitely a first class citizen.

That is good, cause those "configure" stages take for ever :-) 

- Erwin


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] weird ptxdist -q -j go problems

[Michael Olbrich]

Hi,

On Sun, Feb 25, 2018 at 01:02:58PM +0100, Erwin Rol wrote:
> On Sun, 2018-02-25 at 12:43 +0100, Michael Olbrich wrote:
> > > The build system is Centos7 with bash;
> > > GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
> > > 
> > > Not the newest but on the other hand Centos7/RHEL7 is a very stable
> > > platform. It would be very nice if it could be part of the list of
> > > platforms Pengutronix uses to test ptxdist ;-)
> > 
> > I run build tests in several different chroots and Centos7 is one of them.
> > Unfortunately that one is currently broken, because it's missing Python 3.4
> > needed for meson :-/.
> 
> install devtoolset-6, this is what I install in my centos7 docker;
> 
> RUN \
>     echo 'Upgrading all system packages to the latest available versions' >&2 && \
>     yum -y install epel-release && \
>     yum -y install centos-release-scl && \
>     yum -y install devtoolset-6 && \
>     yum -y update
>     
> RUN \
>     echo 'Installing host tools and build system functionality' >&2 && \
>     yum -y install \
[...]

Thanks for the hint. I know that there are multiple ways to install Python
3.4 on Centos7 but I wasn't sure which one I should use. Picking one that
someone already uses in combination with PTXdist sounds like a good idea.
Now I just need to find the time to properly integrate this...

> > Some packages using glib also need host-glib for some of the helper tools
> > like glib-mkenums.
> > 
> > Can you send a patch? And please include the reason in the commit message.
> > Or the build error if it's useful. That helps while reviewing the
> > dependencies for the next update.
> 
> I will have to break it again to get the error msg :-) I'll see if I
> find some time later today.

That would be great, thanks.

> > > BTW: wouldn't it be easier to add select HOST_GLIB to glib.in ? Or even
> > > more general do that for all packages that have a host-* version ?
> > 
> > Not all packages that use glib actually need host-glib and I'd like to
> > avoid unnecessary dependenciesa.
> 
> I'll see if I can find some time to do  
> ./ptxdist clean; ./ptxdist compile pkg_that_uses_glib 
> to see what builds and waht doesn't.
> 
> Of course that is going to cost a lot of build time, so it might take a
> while to get done.

I know. It makes systematic tests like this difficult.

> > > I'll look into it and report back when I know more, certainly now I
> > > know -j really is a first class citizen :-)
> > 
> > I've spent quite some time to make '-j' work without problems and as fast
> > as possible without overloading the build server. All my ptxdist build
> > tests use it. I's most definitely a first class citizen.
> 
> That is good, cause those "configure" stages take for ever :-) 

Exactly, and when I combined -jiX with -jeX then my colleagues complained
when the compile jobs overloaded the build server. So I started looking for
a solution and came up with the mechanism behind -j.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] weird ptxdist -q -j go problems

[Erwin Rol]

Hey Michael,

it happend again, and again with busybox. Maybe busybox also needs some
host tool that isn't build yet, cause I have not seen this error on non
-j builds. 

The logfiles have the following errors;

  LD      printutils/built-in.o
  AR      printutils/lib.a
  HOSTCC  applets/applet_tables
  HOSTCC  applets/usage
  GEN     include/applet_tables.h include/NUM_APPLETS.h
  GEN     include/usage_compressed.h
  GEN     include/applet_tables.h include/NUM_APPLETS.h
  HOSTCC  applets/usage_pod
  CC      applets/applets.o
  LD      applets/built-in.o
  LD      libbb/built-in.o
  CC      libbb/ask_confirmation.o
  CC      libbb/auto_string.o
  CC      libbb/bb_askpass.o
  CC      libbb/bb_bswap_64.o
  CC      libbb/appletlib.o
libbb/appletlib.c: In function 'find_applet_by_name':
libbb/appletlib.c:205:5: warning: "KNOWN_APPNAME_OFFSETS" is not
defined, evaluates to 0 [-Wundef]
 #if KNOWN_APPNAME_OFFSETS <= 0
     ^~~~~~~~~~~~~~~~~~~~~
libbb/appletlib.c:206:8: error: 'NUM_APPLETS' undeclared (first use in
this function); did you mean 'PF_APPLETALK'?
  max = NUM_APPLETS;
        ^~~~~~~~~~~
        PF_APPLETALK
libbb/appletlib.c:206:8: note: each undeclared identifier is reported
only once for each function it appears in
libbb/appletlib.c: At top level:
libbb/appletlib.c:593:29: warning: "NUM_APPLETS" is not defined,
evaluates to 0 [-Wundef]
 # if ENABLE_FEATURE_SUID && NUM_APPLETS > 0
                             ^~~~~~~~~~~
libbb/appletlib.c:879:6: warning: "NUM_APPLETS" is not defined,
evaluates to 0 [-Wundef]
 # if NUM_APPLETS > 0
      ^~~~~~~~~~~
libbb/appletlib.c: In function 'run_applet_and_exit':
libbb/appletlib.c:929:7: warning: "NUM_APPLETS" is not defined,
evaluates to 0 [-Wundef]
 #  if NUM_APPLETS > 0
       ^~~~~~~~~~~
scripts/Makefile.build:197: recipe for target 'libbb/appletlib.o'
failed
make[2]: *** [libbb/appletlib.o] Error 1
make[2]: *** Waiting for unfinished jobs....
  CC      libbb/bb_cat.o
  CC      libbb/bb_do_delay.o
  CC      libbb/change_identity.o
  CC      libbb/bb_pwd.o
  CC      libbb/bb_strtonum.o
  CC      libbb/bb_qsort.o
  CC      libbb/chomp.o
Makefile:742: recipe for target 'libbb' failed
make[1]: *** [libbb] Error 2
make[1]: *** Waiting for unfinished jobs....
  LD      console-tools/built-in.o
  CC      console-tools/chvt.o
  CC      console-tools/deallocvt.o



_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] weird ptxdist -q -j go problems

[Michael Olbrich]

Hi,

On Sun, Feb 25, 2018 at 04:12:04PM +0100, Erwin Rol wrote:
> it happend again, and again with busybox. Maybe busybox also needs some
> host tool that isn't build yet, cause I have not seen this error on non
> -j builds. 
> 
> The logfiles have the following errors;
[...]

I managed to reproduce this once today. I think I figured out what happens.
It's a rather tricky dependency issue. I've pushed a fix to master. Please
test.

Regards,
Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] weird ptxdist -q -j go problems

[Erwin Rol]

Hey Michael,

I have pulled that commit into my tree, and haven't seen the problem
again.

But I must say I didn't see the problem since I reported ist, you know
how it is with race conditions, one time you have them 10 times in a
row the next time it takes weeks to happen again. 

So I will keep an eye on it.

Thanks for the update,

Erwin


On Sat, 2018-03-10 at 01:09 +0100, Michael Olbrich wrote:
> Hi,
> 
> On Sun, Feb 25, 2018 at 04:12:04PM +0100, Erwin Rol wrote:
> > it happend again, and again with busybox. Maybe busybox also needs some
> > host tool that isn't build yet, cause I have not seen this error on non
> > -j builds. 
> > 
> > The logfiles have the following errors;
> 
> [...]
> 
> I managed to reproduce this once today. I think I figured out what happens.
> It's a rather tricky dependency issue. I've pushed a fix to master. Please
> test.
> 
> Regards,
> Michael
> 

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de