Re: [ptxdist] Speed up targetinstall of certain packages

[Christian Melki <christian.melki@t2data.com>]

On 11/4/22 8:12 PM, Alexander Dahl wrote:
> Hello Christian,
> 
> Am Fri, Nov 04, 2022 at 03:37:02PM +0000 schrieb Gieseler, Christian:
>> Hello,
>>
>> I have question regarding the speedup of daily work.
>>
>> We have frontend and backend of our webgui deployed with separate packages. Only task of these package is to deploy the files with
>>
>> @$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/, /var/www/,no)
>>
>> Compile and install stages are empty. The just call targetinfo and touch to skip the stages.
>>
>> The frontend depends on the backend and the backend obviously depends on our application which is called by the backend.
>> So our web-frontend.in file looks like this:
>> ## SECTION=project_specific
>>
>> config WEB_FRONTEND
>> 	bool
>> 	select APP_LAYER
>> 	select WEB_BACKEND
>> 	prompt "e-mode Web Frontend"
>> 	help
>>
>> As expected if i clean and compile APP_LAYER the targetinstallstage of Backend and Frontend are executed again. However this is only a Run-Time only dependency. It is a third-party archive and install_tree takes quite some time even on fast build hosts. Even it if is just a minute it is annoying to spend the time waiting during image creation. Trying to solve that i found "if RUNTIME"  für Run-Time only Dependencys in the documentation here:
>>
>> https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail
>>
>> So my expectation would be that if i change the webfrontend.in file like this:
>>
>> config WEB_FRONTEND
>> 	bool
>> 	select APP_LAYER	if RUNTIME
>> 	select WEB_BACKEND   if RUNTIME
>> 	prompt "e-mode Web Frontend"
>> 	help
> 
> That sounds reasonable and I would have done it the same.
> 
>> The "if RUNTIME" would make sure that the targetinstall stage is not executed again if i just execute a "ptxdist clean app-layer" followed by a "ptxdist images". Same with ptxdist clean root; ptxdist images. It is clear that all targetinstall stages are executed again, but i would expect that the web-frontend is deployed earlier if no build dependency is given.
>>
>> Am i missing something, oder is the "if RUNTIME" Switch not working properly in my ptxdist-2018.12 version? Or does it have no effect on targetinstall stages?
> 
> Not sure how that should behave.  However if you want to speed up the
> build: make sure you call ptxdist with -q or --quiet parameter.  The
> output on screen takes suprisingly much time, even with modern
> terminals, and especially when doing targetinstall of many many files
> (as usually the case with web frontends. been there, done that.)
> 
> Greets
> Alex
> 

I have a slight disagreement here. I don't think the console is slow.
So I did some investigation, mostly since the slowness bugs me too.
I did a:
export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
as time measurement and trace in the shellscript in question (mostly scripts/lib/ptxd_make_xpkg_pkg.sh).
This pretty linear progressive cpu consumption, albeit the big chunks were due to calling of external binaries.

Then I did a timezone package clean and targetinstall.
When targetinstalling a zone there was calls to several binaries.

For one zone I had external calls to (in order):
echo, mkdir, printf(?), flock, ls, rm, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, mkdir, mkdir, mkdir, mkdir, install, install, chmod, chmod, chown, echo.

Some of these could be internal builtins, but the consumed time suggested otherwise.
Either way. Each install took about 26 ms and I could account the majority of that time in forking external programs and waiting for them to return.

So my conclusion is: The whole thing is a bit slow and bash doesn't help.

Regards,
Christian

>>
>> Thanks for any feedback.
>> BR,
>> Christian
>