Re: [ptxdist] Speed up targetinstall of certain packages

[Christian Melki <christian.melki@t2data.com>]

So I did an experiment with bash loadables (builtins).
Turns out bash has a bunch of builtin examples.

I had ptxdist build me a bash-5.2 for x86_64.
With the "loadables" make target.

$ diff -urN ../lib/ptxdist-2022.11.0/rules/bash.make rules/bash.make
--- ../lib/ptxdist-2022.11.0/rules/bash.make	2022-10-21 16:54:50.000000000 +0200
+++ rules/bash.make	2022-11-08 12:51:38.609794625 +0100
@@ -77,6 +77,16 @@
  	--$(call ptx/wwo, PTXCONF_BASH_CURSES)-curses
  
  # ----------------------------------------------------------------------------
+# Compile
+# ----------------------------------------------------------------------------
+
+$(STATEDIR)/bash.compile:
+	@$(call targetinfo)
+	@$(call world/compile, BASH)
+	@$(call compile, BASH, loadables)
+	@$(call touch)
+
+# ----------------------------------------------------------------------------

Since mkdir is slightly less than half of the external binary calls I had it replaced.
I copied the scripts/lib/ptxd_make_xpkg_pkg.sh to my build path
and created a shell-loadables directory there.

$ cp platform-secplatform-x86_64/build-target/bash-5.2/examples/loadables/mkdir scripts/lib/shell-loadables/

$ ls -1  scripts/lib/shell-loadables/
mkdir

$ diff -urN ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh scripts/lib/ptxd_make_xpkg_pkg.sh
--- ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh	2022-10-21 16:54:50.000000000 +0200
+++ scripts/lib/ptxd_make_xpkg_pkg.sh	2022-11-08 13:08:18.074550829 +0100
@@ -682,6 +682,8 @@
  
  ptxd_install_file() {
      local cmd="file"
+    #export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
+    enable -f scripts/lib/shell-loadables/mkdir mkdir
      ptxd_install_file_impl "$@" ||
      ptxd_install_error "install_file failed!"
  }

without mkdir builtin:
finished target timezone.targetinstall.post

real	0m17,043s
user	0m13,522s
sys	0m4,138s

with mkdir builtin:
finished target timezone.targetinstall.post

real	0m12,345s
user	0m9,899s
sys	0m2,916s


On 11/8/22 11:13 AM, Christian Melki wrote:
> 
> 
> On 11/4/22 8:12 PM, Alexander Dahl wrote:
>> Hello Christian,
>>
>> Am Fri, Nov 04, 2022 at 03:37:02PM +0000 schrieb Gieseler, Christian:
>>> Hello,
>>>
>>> I have question regarding the speedup of daily work.
>>>
>>> We have frontend and backend of our webgui deployed with separate packages. Only task of these package is to deploy the files with
>>>
>>> @$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/, /var/www/,no)
>>>
>>> Compile and install stages are empty. The just call targetinfo and touch to skip the stages.
>>>
>>> The frontend depends on the backend and the backend obviously depends on our application which is called by the backend.
>>> So our web-frontend.in file looks like this:
>>> ## SECTION=project_specific
>>>
>>> config WEB_FRONTEND
>>>     bool
>>>     select APP_LAYER
>>>     select WEB_BACKEND
>>>     prompt "e-mode Web Frontend"
>>>     help
>>>
>>> As expected if i clean and compile APP_LAYER the targetinstallstage of Backend and Frontend are executed again. However this is only a Run-Time only dependency. It is a third-party archive and install_tree takes quite some time even on fast build hosts. Even it if is just a minute it is annoying to spend the time waiting during image creation. Trying to solve that i found "if RUNTIME"  für Run-Time only Dependencys in the documentation here:
>>>
>>> https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail
>>>
>>> So my expectation would be that if i change the webfrontend.in file like this:
>>>
>>> config WEB_FRONTEND
>>>     bool
>>>     select APP_LAYER    if RUNTIME
>>>     select WEB_BACKEND   if RUNTIME
>>>     prompt "e-mode Web Frontend"
>>>     help
>>
>> That sounds reasonable and I would have done it the same.
>>
>>> The "if RUNTIME" would make sure that the targetinstall stage is not executed again if i just execute a "ptxdist clean app-layer" followed by a "ptxdist images". Same with ptxdist clean root; ptxdist images. It is clear that all targetinstall stages are executed again, but i would expect that the web-frontend is deployed earlier if no build dependency is given.
>>>
>>> Am i missing something, oder is the "if RUNTIME" Switch not working properly in my ptxdist-2018.12 version? Or does it have no effect on targetinstall stages?
>>
>> Not sure how that should behave.  However if you want to speed up the
>> build: make sure you call ptxdist with -q or --quiet parameter.  The
>> output on screen takes suprisingly much time, even with modern
>> terminals, and especially when doing targetinstall of many many files
>> (as usually the case with web frontends. been there, done that.)
>>
>> Greets
>> Alex
>>
> 
> I have a slight disagreement here. I don't think the console is slow.
> So I did some investigation, mostly since the slowness bugs me too.
> I did a:
> export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
> as time measurement and trace in the shellscript in question (mostly scripts/lib/ptxd_make_xpkg_pkg.sh).
> This pretty linear progressive cpu consumption, albeit the big chunks were due to calling of external binaries.
> 
> Then I did a timezone package clean and targetinstall.
> When targetinstalling a zone there was calls to several binaries.
> 
> For one zone I had external calls to (in order):
> echo, mkdir, printf(?), flock, ls, rm, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, mkdir, mkdir, mkdir, mkdir, install, install, chmod, chmod, chown, echo.
> 
> Some of these could be internal builtins, but the consumed time suggested otherwise.
> Either way. Each install took about 26 ms and I could account the majority of that time in forking external programs and waiting for them to return.
> 
> So my conclusion is: The whole thing is a bit slow and bash doesn't help.
> 
> Regards,
> Christian
> 
>>>
>>> Thanks for any feedback.
>>> BR,
>>> Christian
>>
>