[ptxdist] awk bug in some make dependency scripts?

[ptxdist] awk bug in some make dependency scripts?

[Andreas Bießmann]

Hi all,

I've encountered a possible bug in some of the awk scripts generating
the dependencies. Have not looked deeper into but have this information
so far.

I use ptxdist-2012.07.0, if I do a clean build on a Debian stable (gawk
3.1.7.dfsg-5) the kernel-modules package lack our own modules which
should be placed in /lib/modules/$(uname -r)/extra.
If I build the same stuff with the same ptxdist on the same machine but
use the gawk 4.0.1+dfsg-2~bpo60+1 from debian backports I get the
modules installed successfully.

Did anyone else see this error before? Some pointers where to start
digging into it?

Best regards

Andreas Bießmann

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] awk bug in some make dependency scripts?

[Michael Olbrich]

Hi,

On Fri, Nov 09, 2012 at 03:20:55PM +0100, Andreas Bießmann wrote:
> I've encountered a possible bug in some of the awk scripts generating
> the dependencies. Have not looked deeper into but have this information
> so far.
> 
> I use ptxdist-2012.07.0, if I do a clean build on a Debian stable (gawk
> 3.1.7.dfsg-5) the kernel-modules package lack our own modules which
> should be placed in /lib/modules/$(uname -r)/extra.
> If I build the same stuff with the same ptxdist on the same machine but
> use the gawk 4.0.1+dfsg-2~bpo60+1 from debian backports I get the
> modules installed successfully.
> 
> Did anyone else see this error before? Some pointers where to start
> digging into it?

Is the file platform-<myplat>/state/ptx_dgen_deps.post different?

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] awk bug in some make dependency scripts?

[Andreas Bießmann]

Dear Michael Olbrich,

On 09.11.2012 17:28, Michael Olbrich wrote:
> Hi,
> 
> On Fri, Nov 09, 2012 at 03:20:55PM +0100, Andreas Bießmann wrote:
>> I've encountered a possible bug in some of the awk scripts generating
>> the dependencies. Have not looked deeper into but have this information
>> so far.
>>
>> I use ptxdist-2012.07.0, if I do a clean build on a Debian stable (gawk
>> 3.1.7.dfsg-5) the kernel-modules package lack our own modules which
>> should be placed in /lib/modules/$(uname -r)/extra.
>> If I build the same stuff with the same ptxdist on the same machine but
>> use the gawk 4.0.1+dfsg-2~bpo60+1 from debian backports I get the
>> modules installed successfully.
>>
>> Did anyone else see this error before? Some pointers where to start
>> digging into it?
> 
> Is the file platform-<myplat>/state/ptx_dgen_deps.post different?

yes it is:

---8<---
diff -Nrupa /tmp/ptx_dgen_deps.post.old-awk
/tmp/ptx_dgen_deps.post.new-awk  | diffstat
 ptx_dgen_deps.post.new-awk | 5198
++++++++++++++++++++++-----------------------
 1 file changed, 2599 insertions(+), 2599 deletions(-)
--->8---

On first sight it seems just reordered but will investigate that further.

Best regards

Andreas Bießmann

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] awk bug in some make dependency scripts?

[Michael Olbrich]

Hi,

On Mon, Nov 12, 2012 at 10:02:06AM +0100, Andreas Bießmann wrote:
> On 09.11.2012 17:28, Michael Olbrich wrote:
> > On Fri, Nov 09, 2012 at 03:20:55PM +0100, Andreas Bießmann wrote:
> >> I've encountered a possible bug in some of the awk scripts generating
> >> the dependencies. Have not looked deeper into but have this information
> >> so far.
> >>
> >> I use ptxdist-2012.07.0, if I do a clean build on a Debian stable (gawk
> >> 3.1.7.dfsg-5) the kernel-modules package lack our own modules which
> >> should be placed in /lib/modules/$(uname -r)/extra.
> >> If I build the same stuff with the same ptxdist on the same machine but
> >> use the gawk 4.0.1+dfsg-2~bpo60+1 from debian backports I get the
> >> modules installed successfully.
> >>
> >> Did anyone else see this error before? Some pointers where to start
> >> digging into it?
> > 
> > Is the file platform-<myplat>/state/ptx_dgen_deps.post different?
> 
> yes it is:
> 
> ---8<---
> diff -Nrupa /tmp/ptx_dgen_deps.post.old-awk
> /tmp/ptx_dgen_deps.post.new-awk  | diffstat
>  ptx_dgen_deps.post.new-awk | 5198
> ++++++++++++++++++++++-----------------------
>  1 file changed, 2599 insertions(+), 2599 deletions(-)
> --->8---
> 
> On first sight it seems just reordered but will investigate that further.

Try sorting the files. Most stuff is one-line only anyways and the rest is
some "ifdef" stuff that should still be readable.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] awk bug in some make dependency scripts?

[Andreas Bießmann]

Hi,

On 12.11.2012 12:47, Michael Olbrich wrote:
> Hi,
> 
> On Mon, Nov 12, 2012 at 10:02:06AM +0100, Andreas Bießmann wrote:
>> On 09.11.2012 17:28, Michael Olbrich wrote:
>>> On Fri, Nov 09, 2012 at 03:20:55PM +0100, Andreas Bießmann wrote:
>>>> I've encountered a possible bug in some of the awk scripts generating
>>>> the dependencies. Have not looked deeper into but have this information
>>>> so far.
>>>>
>>>> I use ptxdist-2012.07.0, if I do a clean build on a Debian stable (gawk
>>>> 3.1.7.dfsg-5) the kernel-modules package lack our own modules which
>>>> should be placed in /lib/modules/$(uname -r)/extra.
>>>> If I build the same stuff with the same ptxdist on the same machine but
>>>> use the gawk 4.0.1+dfsg-2~bpo60+1 from debian backports I get the
>>>> modules installed successfully.
>>>>
>>>> Did anyone else see this error before? Some pointers where to start
>>>> digging into it?
>>>
>>> Is the file platform-<myplat>/state/ptx_dgen_deps.post different?
>>
>> yes it is:
>>
>> ---8<---
>> diff -Nrupa /tmp/ptx_dgen_deps.post.old-awk
>> /tmp/ptx_dgen_deps.post.new-awk  | diffstat
>>  ptx_dgen_deps.post.new-awk | 5198
>> ++++++++++++++++++++++-----------------------
>>  1 file changed, 2599 insertions(+), 2599 deletions(-)
>> --->8---
>>
>> On first sight it seems just reordered but will investigate that further.
> 
> Try sorting the files. Most stuff is one-line only anyways and the rest is
> some "ifdef" stuff that should still be readable.

double checked it now. The content of both ptx_dgen_deps.post is the
same after reordering.
I have another strange investigation: I built a smaller testcase to have
faster test results but unfortunately the smaller testcase did always work!
The small Testcase consists only of a kernel plus a simple test module
(and the required host tools). The difference is something like this:

---8<---
% ls -l platform-responder2/state/*.install | wc -l
95
% ls -l platform-test/state/*.install | wc -l
27
--->8---

These following lines show my test-case with the real repository. The
repository is first built completely (so just the rebuild is required).
The rebuild of kernel and datasource-driver is triggered explicitly,
dependency for printer-driver is met implicitly. To show the content of
kernel-modules.ipkg I delete the 'extra' dir in NFS-root before and
check it after the build.
The difference in order of targets built is sticking out and until now
always reproducible (just switch the awk version and do the sequence).

---8<---
with gawk 4.0 (sudo aptitude install gawk/squeeze-backports)
rm -rf platform-responder2/root/lib/modules/3.4.18/extra/
./p drop kernel.compile
./p drop datasource-driver.compile
./p go

target: kernel.compile
target: kernel.install
target: kernel.install.pack
target: kernel.install.post
target: kernel.targetinstall
target: printer-driver.extract.post
target: printer-driver.prepare
target: printer-driver.compile
target: printer-driver.install
target: printer-driver.install.pack
target: printer-driver.install.post
target: printer-driver.targetinstall
target: printer-driver.targetinstall.post
target: datasource-driver.extract.post
target: datasource-driver.prepare
target: datasource-driver.compile
target: datasource-driver.install
target: datasource-driver.install.pack
target: datasource-driver.install.post
target: datasource-driver.targetinstall
target: datasource-driver.targetinstall.post
target: kernel.targetinstall.post

ls -la platform-responder2/root/lib/modules/3.4.18/extra -> OK
----
with gawk 3.1.x (sudo aptitude install gawk/stable)

rm -rf platform-responder2/root/lib/modules/3.4.18/extra/
./p drop kernel.compile
./p drop datasource-driver.compile
./p go

target: kernel.compile
target: kernel.install
target: kernel.install.pack
target: kernel.install.post
target: kernel.targetinstall
target: kernel.targetinstall.post
target: datasource-driver.extract.post
target: datasource-driver.prepare
target: datasource-driver.compile
target: datasource-driver.install
target: datasource-driver.install.pack
target: datasource-driver.install.post
target: datasource-driver.targetinstall
target: datasource-driver.targetinstall.post
target: printer-driver.extract.post
target: printer-driver.prepare
target: printer-driver.compile
target: printer-driver.install
target: printer-driver.install.pack
target: printer-driver.install.post
target: printer-driver.targetinstall
target: printer-driver.targetinstall.post

ls -la platform-responder2/root/lib/modules/3.4.18/extra -> NOT OK
--->8---

Any other pointers?

Best regards

Andreas Bießmann

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] awk bug in some make dependency scripts?

[Michael Olbrich]

Hi,

On Tue, Nov 13, 2012 at 03:35:40PM +0100, Andreas Bießmann wrote:
> On 12.11.2012 12:47, Michael Olbrich wrote:
> > On Mon, Nov 12, 2012 at 10:02:06AM +0100, Andreas Bießmann wrote:
> >> On 09.11.2012 17:28, Michael Olbrich wrote:
> >>> On Fri, Nov 09, 2012 at 03:20:55PM +0100, Andreas Bießmann wrote:
> >>>> I've encountered a possible bug in some of the awk scripts generating
> >>>> the dependencies. Have not looked deeper into but have this information
> >>>> so far.
> >>>>
> >>>> I use ptxdist-2012.07.0, if I do a clean build on a Debian stable (gawk
> >>>> 3.1.7.dfsg-5) the kernel-modules package lack our own modules which
> >>>> should be placed in /lib/modules/$(uname -r)/extra.
> >>>> If I build the same stuff with the same ptxdist on the same machine but
> >>>> use the gawk 4.0.1+dfsg-2~bpo60+1 from debian backports I get the
> >>>> modules installed successfully.
> >>>>
> >>>> Did anyone else see this error before? Some pointers where to start
> >>>> digging into it?
> >>>
> >>> Is the file platform-<myplat>/state/ptx_dgen_deps.post different?
> >>
> >> yes it is:
> >>
> >> ---8<---
> >> diff -Nrupa /tmp/ptx_dgen_deps.post.old-awk
> >> /tmp/ptx_dgen_deps.post.new-awk  | diffstat
> >>  ptx_dgen_deps.post.new-awk | 5198
> >> ++++++++++++++++++++++-----------------------
> >>  1 file changed, 2599 insertions(+), 2599 deletions(-)
> >> --->8---
> >>
> >> On first sight it seems just reordered but will investigate that further.
> > 
> > Try sorting the files. Most stuff is one-line only anyways and the rest is
> > some "ifdef" stuff that should still be readable.
> 
> double checked it now. The content of both ptx_dgen_deps.post is the
> same after reordering.
> I have another strange investigation: I built a smaller testcase to have
> faster test results but unfortunately the smaller testcase did always work!
> The small Testcase consists only of a kernel plus a simple test module
> (and the required host tools). The difference is something like this:
> 
> ---8<---
> % ls -l platform-responder2/state/*.install | wc -l
> 95
> % ls -l platform-test/state/*.install | wc -l
> 27
> --->8---
> 
> These following lines show my test-case with the real repository. The
> repository is first built completely (so just the rebuild is required).
> The rebuild of kernel and datasource-driver is triggered explicitly,
> dependency for printer-driver is met implicitly. To show the content of
> kernel-modules.ipkg I delete the 'extra' dir in NFS-root before and
> check it after the build.
> The difference in order of targets built is sticking out and until now
> always reproducible (just switch the awk version and do the sequence).
> 
> ---8<---
> with gawk 4.0 (sudo aptitude install gawk/squeeze-backports)
> rm -rf platform-responder2/root/lib/modules/3.4.18/extra/
> ./p drop kernel.compile
> ./p drop datasource-driver.compile
> ./p go
> 
> target: kernel.compile
> target: kernel.install
> target: kernel.install.pack
> target: kernel.install.post
> target: kernel.targetinstall
> target: printer-driver.extract.post
> target: printer-driver.prepare
> target: printer-driver.compile
> target: printer-driver.install
> target: printer-driver.install.pack
> target: printer-driver.install.post
> target: printer-driver.targetinstall
> target: printer-driver.targetinstall.post
> target: datasource-driver.extract.post
> target: datasource-driver.prepare
> target: datasource-driver.compile
> target: datasource-driver.install
> target: datasource-driver.install.pack
> target: datasource-driver.install.post
> target: datasource-driver.targetinstall
> target: datasource-driver.targetinstall.post
> target: kernel.targetinstall.post
> 
> ls -la platform-responder2/root/lib/modules/3.4.18/extra -> OK
> ----
> with gawk 3.1.x (sudo aptitude install gawk/stable)
> 
> rm -rf platform-responder2/root/lib/modules/3.4.18/extra/
> ./p drop kernel.compile
> ./p drop datasource-driver.compile
> ./p go
> 
> target: kernel.compile
> target: kernel.install
> target: kernel.install.pack
> target: kernel.install.post
> target: kernel.targetinstall
> target: kernel.targetinstall.post
> target: datasource-driver.extract.post
> target: datasource-driver.prepare
> target: datasource-driver.compile
> target: datasource-driver.install
> target: datasource-driver.install.pack
> target: datasource-driver.install.post
> target: datasource-driver.targetinstall
> target: datasource-driver.targetinstall.post
> target: printer-driver.extract.post
> target: printer-driver.prepare
> target: printer-driver.compile
> target: printer-driver.install
> target: printer-driver.install.pack
> target: printer-driver.install.post
> target: printer-driver.targetinstall
> target: printer-driver.targetinstall.post
> 
> ls -la platform-responder2/root/lib/modules/3.4.18/extra -> NOT OK
> --->8---
> 
> Any other pointers?

Hmmm, what order do you get if you run "./p targetinstall kernel" instead
of "./p go"?

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] awk bug in some make dependency scripts?

[Andreas Bießmann]

On 13.11.2012 21:02, Michael Olbrich wrote:
> Hi,
> 
> On Tue, Nov 13, 2012 at 03:35:40PM +0100, Andreas Bießmann wrote:
>> On 12.11.2012 12:47, Michael Olbrich wrote:
>>> On Mon, Nov 12, 2012 at 10:02:06AM +0100, Andreas Bießmann wrote:
>>>> On 09.11.2012 17:28, Michael Olbrich wrote:
>>>>> On Fri, Nov 09, 2012 at 03:20:55PM +0100, Andreas Bießmann wrote:

<snip>

>> These following lines show my test-case with the real repository. The
>> repository is first built completely (so just the rebuild is required).
>> The rebuild of kernel and datasource-driver is triggered explicitly,
>> dependency for printer-driver is met implicitly. To show the content of
>> kernel-modules.ipkg I delete the 'extra' dir in NFS-root before and
>> check it after the build.
>> The difference in order of targets built is sticking out and until now
>> always reproducible (just switch the awk version and do the sequence).
>>
>> ---8<---
>> with gawk 4.0 (sudo aptitude install gawk/squeeze-backports)
>> rm -rf platform-responder2/root/lib/modules/3.4.18/extra/
>> ./p drop kernel.compile
>> ./p drop datasource-driver.compile
>> ./p go
>>
>> target: kernel.compile
>> target: kernel.install
>> target: kernel.install.pack
>> target: kernel.install.post
>> target: kernel.targetinstall
>> target: printer-driver.extract.post
>> target: printer-driver.prepare
>> target: printer-driver.compile
>> target: printer-driver.install
>> target: printer-driver.install.pack
>> target: printer-driver.install.post
>> target: printer-driver.targetinstall
>> target: printer-driver.targetinstall.post
>> target: datasource-driver.extract.post
>> target: datasource-driver.prepare
>> target: datasource-driver.compile
>> target: datasource-driver.install
>> target: datasource-driver.install.pack
>> target: datasource-driver.install.post
>> target: datasource-driver.targetinstall
>> target: datasource-driver.targetinstall.post
>> target: kernel.targetinstall.post
>>
>> ls -la platform-responder2/root/lib/modules/3.4.18/extra -> OK
>> ----
>> with gawk 3.1.x (sudo aptitude install gawk/stable)
>>
>> rm -rf platform-responder2/root/lib/modules/3.4.18/extra/
>> ./p drop kernel.compile
>> ./p drop datasource-driver.compile
>> ./p go
>>
>> target: kernel.compile
>> target: kernel.install
>> target: kernel.install.pack
>> target: kernel.install.post
>> target: kernel.targetinstall
>> target: kernel.targetinstall.post
>> target: datasource-driver.extract.post
>> target: datasource-driver.prepare
>> target: datasource-driver.compile
>> target: datasource-driver.install
>> target: datasource-driver.install.pack
>> target: datasource-driver.install.post
>> target: datasource-driver.targetinstall
>> target: datasource-driver.targetinstall.post
>> target: printer-driver.extract.post
>> target: printer-driver.prepare
>> target: printer-driver.compile
>> target: printer-driver.install
>> target: printer-driver.install.pack
>> target: printer-driver.install.post
>> target: printer-driver.targetinstall
>> target: printer-driver.targetinstall.post
>>
>> ls -la platform-responder2/root/lib/modules/3.4.18/extra -> NOT OK
>> --->8---
>>
>> Any other pointers?
> 
> Hmmm, what order do you get if you run "./p targetinstall kernel" instead
> of "./p go"?

Well, does not build the drivers at all. And it is explicable cause
(shame on me) I missed the explicit dependency kernel.targetinstall.port
-> driver.targetinstall. My small example, however, had this rule so I
could not reproduce the behaviour with it.
Many thanks to you Michael and Tim Niemeyer who pushed me in the right
direction.

I wonder if we can add an implicit rule to build <package>.targetinstall
before kernel.targetinstall.post if we depend on KERNEL. So we could get
rid of the explicit rule in a driver.

Best regards

Andreas Bießmann

-- 
ptxdist mailing list
ptxdist@pengutronix.de