[ptxdist] Loss of speedup for subsequent different -but still similar- images build

[ptxdist] Loss of speedup for subsequent different -but still similar- images build

[Henzer, Kevin]

Hello !

Thanks for adding me to the mailing list, and thank for this nice tool that PTXdist is !

I'm new to PTXdist, and I have inherited a build setup that creates two different but quite similar system images, following two different global configurations (config_1 and config_2). 
One is the rescue update image of the system, the other the full-featured image. It is using a script to build them both in a row. 

Before a recent update, calling the build process successively on those two configurations gave a big speed-up for the second build. The system was reusing most of the packages.

./ptxdist go --ptxconfig=configs/config_1 --platformconfig=configs/platformconfig 	# takes time only the first time, else quite fast
./ptxdist images --ptxconfig=configs/ config_1  --platformconfig=configs/platformconfig  # quite fast

./ptxdist go --ptxconfig=configs/config_2 --platformconfig=configs/platformconfig 	# quite fast
./ptxdist images --ptxconfig=configs/ config_2  --platformconfig=configs/platformconfig  # quite fast



But after updating PTXdist to newer version (now using PTXdist version 2021.03.0), and updating the config files, it seems something has changed, and the outcome is different. 

./ptxdist go --ptxconfig=configs/config_1 --platformconfig=configs/platformconfig 	# takes time at each build
./ptxdist images --ptxconfig=configs/ config_1  --platformconfig=configs/platformconfig  # quite fast

./ptxdist go --ptxconfig=configs/config_2 --platformconfig=configs/platformconfig 	# takes time at each build, seems the system is rebuilding everything


And I got this as first output when switching from one configuration to the other:

-----------------------------------------------------------------
target: host-system-perl.[some hash].cfghash
-----------------------------------------------------------------

Configuration changed! Reconfiguring...



I checked this package and seems to me it is just ensuring that Perl is installed on the host and that the right features are available.
I checked all config options that I think are linked to this package (I searched in rules/host-system-perl.make) : 

PTXCONF_HOST_SYSTEM_PERL_BUILD
PTXCONF_HOST_SYSTEM_PERL_JSON
PTXCONF_HOST_SYSTEM_PERL_LOCALE_PO
PTXCONF_HOST_SYSTEM_PERL_XMLPARSER
PTXCONF_HOST_SYSTEM_PERL_YAPP

And they are set the same in both config files.


Still, the hash of this package is changing between each call with different config. 

So, I was wondering:
- Can it be that the reconfiguration and "rebuild" of this package is what triggers the full image rebuild?  
- If yes, what could trigger this host-system-perl reconfiguration, in your opinion ? 


And most important for me, how are those .cfghash computed ? If you have a quick link to the relevant part of the doc, I'll be very happy !

In advance, thanks !

Regards,

Kevin Henzer
   



-----------------------------
Eaton Industries Manufacturing GmbH ~ Registered place of business: Route de la Longeraie 7, 1110, Morges, Switzerland 

-----------------------------


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de

Re: [ptxdist] Loss of speedup for subsequent different -but still similar- images build

[Ian Abbott]

On 23/09/2021 14:02, Henzer, Kevin wrote:
> Hello !
> 
> Thanks for adding me to the mailing list, and thank for this nice tool that PTXdist is !
> 
> I'm new to PTXdist, and I have inherited a build setup that creates two different but quite similar system images, following two different global configurations (config_1 and config_2).
> One is the rescue update image of the system, the other the full-featured image. It is using a script to build them both in a row.
> 
> Before a recent update, calling the build process successively on those two configurations gave a big speed-up for the second build. The system was reusing most of the packages.
> 
> ./ptxdist go --ptxconfig=configs/config_1 --platformconfig=configs/platformconfig 	# takes time only the first time, else quite fast
> ./ptxdist images --ptxconfig=configs/ config_1  --platformconfig=configs/platformconfig  # quite fast
> 
> ./ptxdist go --ptxconfig=configs/config_2 --platformconfig=configs/platformconfig 	# quite fast
> ./ptxdist images --ptxconfig=configs/ config_2  --platformconfig=configs/platformconfig  # quite fast
> 
> 
> 
> But after updating PTXdist to newer version (now using PTXdist version 2021.03.0), and updating the config files, it seems something has changed, and the outcome is different.
> 
> ./ptxdist go --ptxconfig=configs/config_1 --platformconfig=configs/platformconfig 	# takes time at each build
> ./ptxdist images --ptxconfig=configs/ config_1  --platformconfig=configs/platformconfig  # quite fast
> 
> ./ptxdist go --ptxconfig=configs/config_2 --platformconfig=configs/platformconfig 	# takes time at each build, seems the system is rebuilding everything
> 

This is not an answer to the problem of possibly unnecessary rebuilding, 
but you can reduce the build time considerably using the -j option to 
build packages in parallel.  (It has an optional parameter to set the 
number of jobs to run in parallel, but the default is based on the 
number of available CPUs.)  That option is best combined with the -q 
(--quiet) option to make the build messages on the terminal less 
confusing, otherwise the parallel build messages on the terminal would 
be a chaotic mess.

-- 
-=( Ian Abbott <abbotti@mev.co.uk> || MEV Ltd. is a company  )=-
-=( registered in England & Wales.  Regd. number: 02862268.  )=-
-=( Regd. addr.: S11 & 12 Building 67, Europa Business Park, )=-
-=( Bird Hall Lane, STOCKPORT, SK3 0XA, UK. || www.mev.co.uk )=-

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de

Re: [ptxdist] Loss of speedup for subsequent different -but still similar- images build

[Juergen Borleis]

Hi Kevin,

Am Donnerstag, dem 23.09.2021 um 13:02 +0000 schrieb Henzer, Kevin:
> […]
> ./ptxdist go --ptxconfig=configs/config_1 --platformconfig=configs/platformconfig       # takes time only the first time, else quite fast
> ./ptxdist images --ptxconfig=configs/ config_1  --platformconfig=configs/platformconfig  # quite fast
> 
> ./ptxdist go --ptxconfig=configs/config_2 --platformconfig=configs/platformconfig       # quite fast
> ./ptxdist images --ptxconfig=configs/ config_2  --platformconfig=configs/platformconfig  # quite fast

What is the difference between 'configs/config_1' and 'configs/config_2'? Maybe
they differ in the setting of some kind of base package and then always all
dependencies must be rebuilt.

jb

-- 
Pengutronix e.K.                       | Juergen Borleis             |
Steuerwalder Str. 21                   | https://www.pengutronix.de/ |
31137 Hildesheim, Germany              | Phone: +49-5121-206917-128 |
Amtsgericht Hildesheim, HRA 2686       | Fax:   +49-5121-206917-9 |


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de

Re: [ptxdist] Loss of speedup for subsequent different -but still similar- images build

[Michael Olbrich]

Hi,

On Thu, Sep 23, 2021 at 01:02:20PM +0000, Henzer, Kevin wrote:
> Thanks for adding me to the mailing list, and thank for this nice tool that PTXdist is !
> 
> I'm new to PTXdist, and I have inherited a build setup that creates two different but quite similar system images, following two different global configurations (config_1 and config_2). 
> One is the rescue update image of the system, the other the full-featured image. It is using a script to build them both in a row. 
> 
> Before a recent update, calling the build process successively on those two configurations gave a big speed-up for the second build. The system was reusing most of the packages.
> 
> ./ptxdist go --ptxconfig=configs/config_1 --platformconfig=configs/platformconfig 	# takes time only the first time, else quite fast
> ./ptxdist images --ptxconfig=configs/ config_1  --platformconfig=configs/platformconfig  # quite fast
> 
> ./ptxdist go --ptxconfig=configs/config_2 --platformconfig=configs/platformconfig 	# quite fast
> ./ptxdist images --ptxconfig=configs/ config_2  --platformconfig=configs/platformconfig  # quite fast
> 
> 
> 
> But after updating PTXdist to newer version (now using PTXdist version 2021.03.0), and updating the config files, it seems something has changed, and the outcome is different. 
> 
> ./ptxdist go --ptxconfig=configs/config_1 --platformconfig=configs/platformconfig 	# takes time at each build
> ./ptxdist images --ptxconfig=configs/ config_1  --platformconfig=configs/platformconfig  # quite fast
> 
> ./ptxdist go --ptxconfig=configs/config_2 --platformconfig=configs/platformconfig 	# takes time at each build, seems the system is rebuilding everything
> 
> 
> And I got this as first output when switching from one configuration to the other:
> 
> -----------------------------------------------------------------
> target: host-system-perl.[some hash].cfghash
> -----------------------------------------------------------------
> 
> Configuration changed! Reconfiguring...

This is most likely the cause. Just about everything in the BSP depends
(indirectly) on this.

> I checked this package and seems to me it is just ensuring that Perl is installed on the host and that the right features are available.
> I checked all config options that I think are linked to this package (I searched in rules/host-system-perl.make) : 
> 
> PTXCONF_HOST_SYSTEM_PERL_BUILD
> PTXCONF_HOST_SYSTEM_PERL_JSON
> PTXCONF_HOST_SYSTEM_PERL_LOCALE_PO
> PTXCONF_HOST_SYSTEM_PERL_XMLPARSER
> PTXCONF_HOST_SYSTEM_PERL_YAPP
> 
> And they are set the same in both config files.
> 
> 
> Still, the hash of this package is changing between each call with different config. 
> 
> So, I was wondering:
> - Can it be that the reconfiguration and "rebuild" of this package is
> what triggers the full image rebuild?

Yes, definitely.

> - If yes, what could trigger this host-system-perl reconfiguration, in your opinion ? 

I'm not sure. It look like it shouldn't change, so lets dig deeper.

> And most important for me, how are those .cfghash computed ? If you have a quick link to the relevant part of the doc, I'll be very happy !

So, there is no documentation and it changed a bit over time. PTXdist
creates a file for this and calculates the md5. The file contains:
 - the content of host-system-perl.make and host-system-perl.in
 - all PTXCONF_HOST_SYSTEM_PERL* options

While PTXdist is running, this file can be found at
/tmp/ptxdist.XXXXXX/pkghash-HOST_SYSTEM_PERL (XXXXXX is some random string
here). Compare the two versions to see what's different.

Michael


-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de