Re: [ptxdist] Speed up targetinstall of certain packages

[Christian Melki <christian.melki@t2data.com>]

On 11/11/22 17:08, Michael Olbrich wrote:
> Hi,
> 
> On Tue, Nov 08, 2022 at 01:10:32PM +0100, Christian Melki wrote:
>> So I did an experiment with bash loadables (builtins).
>> Turns out bash has a bunch of builtin examples.
>>
>> I had ptxdist build me a bash-5.2 for x86_64.
>> With the "loadables" make target.
>>
>> $ diff -urN ../lib/ptxdist-2022.11.0/rules/bash.make rules/bash.make
>> --- ../lib/ptxdist-2022.11.0/rules/bash.make	2022-10-21 16:54:50.000000000 +0200
>> +++ rules/bash.make	2022-11-08 12:51:38.609794625 +0100
>> @@ -77,6 +77,16 @@
>>  	--$(call ptx/wwo, PTXCONF_BASH_CURSES)-curses
>>  # ----------------------------------------------------------------------------
>> +# Compile
>> +# ----------------------------------------------------------------------------
>> +
>> +$(STATEDIR)/bash.compile:
>> +	@$(call targetinfo)
>> +	@$(call world/compile, BASH)
>> +	@$(call compile, BASH, loadables)
>> +	@$(call touch)
>> +
>> +# ----------------------------------------------------------------------------
>>
>> Since mkdir is slightly less than half of the external binary calls I had it replaced.
>> I copied the scripts/lib/ptxd_make_xpkg_pkg.sh to my build path
>> and created a shell-loadables directory there.
>>
>> $ cp platform-secplatform-x86_64/build-target/bash-5.2/examples/loadables/mkdir scripts/lib/shell-loadables/
>>
>> $ ls -1  scripts/lib/shell-loadables/
>> mkdir
>>
>> $ diff -urN ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh scripts/lib/ptxd_make_xpkg_pkg.sh
>> --- ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh	2022-10-21 16:54:50.000000000 +0200
>> +++ scripts/lib/ptxd_make_xpkg_pkg.sh	2022-11-08 13:08:18.074550829 +0100
>> @@ -682,6 +682,8 @@
>>  ptxd_install_file() {
>>      local cmd="file"
>> +    #export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
>> +    enable -f scripts/lib/shell-loadables/mkdir mkdir
>>      ptxd_install_file_impl "$@" ||
>>      ptxd_install_error "install_file failed!"
>>  }
>>
>> without mkdir builtin:
>> finished target timezone.targetinstall.post
>>
>> real	0m17,043s
>> user	0m13,522s
>> sys	0m4,138s
>>
>> with mkdir builtin:
>> finished target timezone.targetinstall.post
>>
>> real	0m12,345s
>> user	0m9,899s
>> sys	0m2,916s
> 
> So loadable builtins in bash is interesting, but not something we can rely
> on at this time. But this whole thing got me thinking. Most mkdirs are
> don't actually do anything. They are just in case the directory does not
> exist yet. So we can do that:
> 
> if [ ! -d  "${dir}" ]; then
> 	mkdir -p "${dir}"
> fi
> 
> That avoid most of the mkdirs and the speedup is considerably. Similar to
> the builtin I think.  And I found some more things to improve:
> 
> The 'flock' is only needed when building packages in parallel, so we can
> skip it for things like "ptxdist drop foo.compile; ptxdist targetinstall
> foo", and that's where the slow targetinstall hurts the moset.
> 
> Do the chmod/chown with the 'install' when possible.
> 
> And some stuff to improve the ptxdist startup time.
> 
> For "time ptxdist targetinstall timezone" I get:
> 
> Before:
> real    0m29.167s
> user    0m19.389s
> sys     0m9.903s
> 
> After:
> real    0m8.887s
> user    0m5.954s
> sys     0m2.470s
> 
> This stuff should hit master pretty soon.
> 
> Thanks for the inspiration :-).
> 
> Michael
> 

Super!
The end goal was reached either way.
A faster ptxdist! Wee! :D

And thanks for the kind words,
Christian

>> On 11/8/22 11:13 AM, Christian Melki wrote:
>>>
>>>
>>> On 11/4/22 8:12 PM, Alexander Dahl wrote:
>>>> Hello Christian,
>>>>
>>>> Am Fri, Nov 04, 2022 at 03:37:02PM +0000 schrieb Gieseler, Christian:
>>>>> Hello,
>>>>>
>>>>> I have question regarding the speedup of daily work.
>>>>>
>>>>> We have frontend and backend of our webgui deployed with separate packages. Only task of these package is to deploy the files with
>>>>>
>>>>> @$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/, /var/www/,no)
>>>>>
>>>>> Compile and install stages are empty. The just call targetinfo and touch to skip the stages.
>>>>>
>>>>> The frontend depends on the backend and the backend obviously depends on our application which is called by the backend.
>>>>> So our web-frontend.in file looks like this:
>>>>> ## SECTION=project_specific
>>>>>
>>>>> config WEB_FRONTEND
>>>>>     bool
>>>>>     select APP_LAYER
>>>>>     select WEB_BACKEND
>>>>>     prompt "e-mode Web Frontend"
>>>>>     help
>>>>>
>>>>> As expected if i clean and compile APP_LAYER the targetinstallstage of Backend and Frontend are executed again. However this is only a Run-Time only dependency. It is a third-party archive and install_tree takes quite some time even on fast build hosts. Even it if is just a minute it is annoying to spend the time waiting during image creation. Trying to solve that i found "if RUNTIME"  für Run-Time only Dependencys in the documentation here:
>>>>>
>>>>> https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail
>>>>>
>>>>> So my expectation would be that if i change the webfrontend.in file like this:
>>>>>
>>>>> config WEB_FRONTEND
>>>>>     bool
>>>>>     select APP_LAYER    if RUNTIME
>>>>>     select WEB_BACKEND   if RUNTIME
>>>>>     prompt "e-mode Web Frontend"
>>>>>     help
>>>>
>>>> That sounds reasonable and I would have done it the same.
>>>>
>>>>> The "if RUNTIME" would make sure that the targetinstall stage is not executed again if i just execute a "ptxdist clean app-layer" followed by a "ptxdist images". Same with ptxdist clean root; ptxdist images. It is clear that all targetinstall stages are executed again, but i would expect that the web-frontend is deployed earlier if no build dependency is given.
>>>>>
>>>>> Am i missing something, oder is the "if RUNTIME" Switch not working properly in my ptxdist-2018.12 version? Or does it have no effect on targetinstall stages?
>>>>
>>>> Not sure how that should behave.  However if you want to speed up the
>>>> build: make sure you call ptxdist with -q or --quiet parameter.  The
>>>> output on screen takes suprisingly much time, even with modern
>>>> terminals, and especially when doing targetinstall of many many files
>>>> (as usually the case with web frontends. been there, done that.)
>>>>
>>>> Greets
>>>> Alex
>>>>
>>>
>>> I have a slight disagreement here. I don't think the console is slow.
>>> So I did some investigation, mostly since the slowness bugs me too.
>>> I did a:
>>> export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
>>> as time measurement and trace in the shellscript in question (mostly scripts/lib/ptxd_make_xpkg_pkg.sh).
>>> This pretty linear progressive cpu consumption, albeit the big chunks were due to calling of external binaries.
>>>
>>> Then I did a timezone package clean and targetinstall.
>>> When targetinstalling a zone there was calls to several binaries.
>>>
>>> For one zone I had external calls to (in order):
>>> echo, mkdir, printf(?), flock, ls, rm, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, mkdir, mkdir, mkdir, mkdir, install, install, chmod, chmod, chown, echo.
>>>
>>> Some of these could be internal builtins, but the consumed time suggested otherwise.
>>> Either way. Each install took about 26 ms and I could account the majority of that time in forking external programs and waiting for them to return.
>>>
>>> So my conclusion is: The whole thing is a bit slow and bash doesn't help.
>>>
>>> Regards,
>>> Christian
>>>
>>>>>
>>>>> Thanks for any feedback.
>>>>> BR,
>>>>> Christian
>>>>
>>>
>>
>>
>