* [ptxdist] Speed up targetinstall of certain packages
@ 2022-11-04 15:37 Gieseler, Christian
2022-11-04 19:12 ` Alexander Dahl
2022-11-08 8:25 ` Michael Olbrich
0 siblings, 2 replies; 8+ messages in thread
From: Gieseler, Christian @ 2022-11-04 15:37 UTC (permalink / raw)
To: 'ptxdist@pengutronix.de'
Hello,
I have question regarding the speedup of daily work.
We have frontend and backend of our webgui deployed with separate packages. Only task of these package is to deploy the files with
@$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/, /var/www/,no)
Compile and install stages are empty. The just call targetinfo and touch to skip the stages.
The frontend depends on the backend and the backend obviously depends on our application which is called by the backend.
So our web-frontend.in file looks like this:
## SECTION=project_specific
config WEB_FRONTEND
bool
select APP_LAYER
select WEB_BACKEND
prompt "e-mode Web Frontend"
help
As expected if i clean and compile APP_LAYER the targetinstallstage of Backend and Frontend are executed again. However this is only a Run-Time only dependency. It is a third-party archive and install_tree takes quite some time even on fast build hosts. Even it if is just a minute it is annoying to spend the time waiting during image creation. Trying to solve that i found "if RUNTIME" für Run-Time only Dependencys in the documentation here:
https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail
So my expectation would be that if i change the webfrontend.in file like this:
config WEB_FRONTEND
bool
select APP_LAYER if RUNTIME
select WEB_BACKEND if RUNTIME
prompt "e-mode Web Frontend"
help
The "if RUNTIME" would make sure that the targetinstall stage is not executed again if i just execute a "ptxdist clean app-layer" followed by a "ptxdist images". Same with ptxdist clean root; ptxdist images. It is clear that all targetinstall stages are executed again, but i would expect that the web-frontend is deployed earlier if no build dependency is given.
Am i missing something, oder is the "if RUNTIME" Switch not working properly in my ptxdist-2018.12 version? Or does it have no effect on targetinstall stages?
Thanks for any feedback.
BR,
Christian
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ptxdist] Speed up targetinstall of certain packages
2022-11-04 15:37 [ptxdist] Speed up targetinstall of certain packages Gieseler, Christian
@ 2022-11-04 19:12 ` Alexander Dahl
2022-11-08 10:13 ` Christian Melki
2022-11-08 8:25 ` Michael Olbrich
1 sibling, 1 reply; 8+ messages in thread
From: Alexander Dahl @ 2022-11-04 19:12 UTC (permalink / raw)
To: ptxdist; +Cc: Gieseler, Christian
[-- Attachment #1: Type: text/plain, Size: 2910 bytes --]
Hello Christian,
Am Fri, Nov 04, 2022 at 03:37:02PM +0000 schrieb Gieseler, Christian:
> Hello,
>
> I have question regarding the speedup of daily work.
>
> We have frontend and backend of our webgui deployed with separate packages. Only task of these package is to deploy the files with
>
> @$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/, /var/www/,no)
>
> Compile and install stages are empty. The just call targetinfo and touch to skip the stages.
>
> The frontend depends on the backend and the backend obviously depends on our application which is called by the backend.
> So our web-frontend.in file looks like this:
> ## SECTION=project_specific
>
> config WEB_FRONTEND
> bool
> select APP_LAYER
> select WEB_BACKEND
> prompt "e-mode Web Frontend"
> help
>
> As expected if i clean and compile APP_LAYER the targetinstallstage of Backend and Frontend are executed again. However this is only a Run-Time only dependency. It is a third-party archive and install_tree takes quite some time even on fast build hosts. Even it if is just a minute it is annoying to spend the time waiting during image creation. Trying to solve that i found "if RUNTIME" für Run-Time only Dependencys in the documentation here:
>
> https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail
>
> So my expectation would be that if i change the webfrontend.in file like this:
>
> config WEB_FRONTEND
> bool
> select APP_LAYER if RUNTIME
> select WEB_BACKEND if RUNTIME
> prompt "e-mode Web Frontend"
> help
That sounds reasonable and I would have done it the same.
> The "if RUNTIME" would make sure that the targetinstall stage is not executed again if i just execute a "ptxdist clean app-layer" followed by a "ptxdist images". Same with ptxdist clean root; ptxdist images. It is clear that all targetinstall stages are executed again, but i would expect that the web-frontend is deployed earlier if no build dependency is given.
>
> Am i missing something, oder is the "if RUNTIME" Switch not working properly in my ptxdist-2018.12 version? Or does it have no effect on targetinstall stages?
Not sure how that should behave. However if you want to speed up the
build: make sure you call ptxdist with -q or --quiet parameter. The
output on screen takes suprisingly much time, even with modern
terminals, and especially when doing targetinstall of many many files
(as usually the case with web frontends. been there, done that.)
Greets
Alex
>
> Thanks for any feedback.
> BR,
> Christian
--
/"\ ASCII RIBBON | »With the first link, the chain is forged. The first
\ / CAMPAIGN | speech censured, the first thought forbidden, the
X AGAINST | first freedom denied, chains us all irrevocably.«
/ \ HTML MAIL | (Jean-Luc Picard, quoting Judge Aaron Satie)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ptxdist] Speed up targetinstall of certain packages
2022-11-04 15:37 [ptxdist] Speed up targetinstall of certain packages Gieseler, Christian
2022-11-04 19:12 ` Alexander Dahl
@ 2022-11-08 8:25 ` Michael Olbrich
1 sibling, 0 replies; 8+ messages in thread
From: Michael Olbrich @ 2022-11-08 8:25 UTC (permalink / raw)
To: Gieseler, Christian; +Cc: 'ptxdist@pengutronix.de'
Hi,
On Fri, Nov 04, 2022 at 03:37:02PM +0000, Gieseler, Christian wrote:
> I have question regarding the speedup of daily work.
>
> We have frontend and backend of our webgui deployed with separate packages. Only task of these package is to deploy the files with
>
> @$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/, /var/www/,no)
>
> Compile and install stages are empty. The just call targetinfo and touch to skip the stages.
>
> The frontend depends on the backend and the backend obviously depends on our application which is called by the backend.
> So our web-frontend.in file looks like this:
> ## SECTION=project_specific
>
> config WEB_FRONTEND
> bool
> select APP_LAYER
> select WEB_BACKEND
> prompt "e-mode Web Frontend"
> help
>
> As expected if i clean and compile APP_LAYER the targetinstallstage of
> Backend and Frontend are executed again. However this is only a Run-Time
> only dependency. It is a third-party archive and install_tree takes quite
> some time even on fast build hosts. Even it if is just a minute it is
> annoying to spend the time waiting during image creation.
I know. I've tried to speed this up in the past but there are multiple
binaries that are executed for each file and that takes time. I've found no
good way to avoid this.
> Trying to solve
> that i found "if RUNTIME" für Run-Time only Dependencys in the
> documentation here:
>
> https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail
>
> So my expectation would be that if i change the webfrontend.in file like this:
>
> config WEB_FRONTEND
> bool
> select APP_LAYER if RUNTIME
> select WEB_BACKEND if RUNTIME
> prompt "e-mode Web Frontend"
> help
>
> The "if RUNTIME" would make sure that the targetinstall stage is not
> executed again if i just execute a "ptxdist clean app-layer" followed by
> a "ptxdist images". Same with ptxdist clean root; ptxdist images. It is
> clear that all targetinstall stages are executed again, but i would
> expect that the web-frontend is deployed earlier if no build dependency
> is given.
>
> Am i missing something, oder is the "if RUNTIME" Switch not working
> properly in my ptxdist-2018.12 version? Or does it have no effect on
> targetinstall stages?
A 'if RUNTIME' dependency still includes a dependency between targetinstall
stages. This happen because the generated opkg include a 'Depends:' on the
other package. And the opkg package name is only generated during the
targetinstall stage. So the targetinstall dependency is necessary to make
this work.
Hmm, if we just don't create the 'Depends:' in the opkg, then we could
avoid the dependency. Maybe make it optional? Or only create if the opkg
tool is actually installed on the target? Ideas and patches welcome.
Michael
--
Pengutronix e.K. | |
Steuerwalder Str. 21 | http://www.pengutronix.de/ |
31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ptxdist] Speed up targetinstall of certain packages
2022-11-04 19:12 ` Alexander Dahl
@ 2022-11-08 10:13 ` Christian Melki
2022-11-08 12:10 ` Christian Melki
2022-11-08 12:50 ` Alexander Dahl
0 siblings, 2 replies; 8+ messages in thread
From: Christian Melki @ 2022-11-08 10:13 UTC (permalink / raw)
To: post, CG, Michael Olbrich; +Cc: ptxdist
On 11/4/22 8:12 PM, Alexander Dahl wrote:
> Hello Christian,
>
> Am Fri, Nov 04, 2022 at 03:37:02PM +0000 schrieb Gieseler, Christian:
>> Hello,
>>
>> I have question regarding the speedup of daily work.
>>
>> We have frontend and backend of our webgui deployed with separate packages. Only task of these package is to deploy the files with
>>
>> @$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/, /var/www/,no)
>>
>> Compile and install stages are empty. The just call targetinfo and touch to skip the stages.
>>
>> The frontend depends on the backend and the backend obviously depends on our application which is called by the backend.
>> So our web-frontend.in file looks like this:
>> ## SECTION=project_specific
>>
>> config WEB_FRONTEND
>> bool
>> select APP_LAYER
>> select WEB_BACKEND
>> prompt "e-mode Web Frontend"
>> help
>>
>> As expected if i clean and compile APP_LAYER the targetinstallstage of Backend and Frontend are executed again. However this is only a Run-Time only dependency. It is a third-party archive and install_tree takes quite some time even on fast build hosts. Even it if is just a minute it is annoying to spend the time waiting during image creation. Trying to solve that i found "if RUNTIME" für Run-Time only Dependencys in the documentation here:
>>
>> https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail
>>
>> So my expectation would be that if i change the webfrontend.in file like this:
>>
>> config WEB_FRONTEND
>> bool
>> select APP_LAYER if RUNTIME
>> select WEB_BACKEND if RUNTIME
>> prompt "e-mode Web Frontend"
>> help
>
> That sounds reasonable and I would have done it the same.
>
>> The "if RUNTIME" would make sure that the targetinstall stage is not executed again if i just execute a "ptxdist clean app-layer" followed by a "ptxdist images". Same with ptxdist clean root; ptxdist images. It is clear that all targetinstall stages are executed again, but i would expect that the web-frontend is deployed earlier if no build dependency is given.
>>
>> Am i missing something, oder is the "if RUNTIME" Switch not working properly in my ptxdist-2018.12 version? Or does it have no effect on targetinstall stages?
>
> Not sure how that should behave. However if you want to speed up the
> build: make sure you call ptxdist with -q or --quiet parameter. The
> output on screen takes suprisingly much time, even with modern
> terminals, and especially when doing targetinstall of many many files
> (as usually the case with web frontends. been there, done that.)
>
> Greets
> Alex
>
I have a slight disagreement here. I don't think the console is slow.
So I did some investigation, mostly since the slowness bugs me too.
I did a:
export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
as time measurement and trace in the shellscript in question (mostly scripts/lib/ptxd_make_xpkg_pkg.sh).
This pretty linear progressive cpu consumption, albeit the big chunks were due to calling of external binaries.
Then I did a timezone package clean and targetinstall.
When targetinstalling a zone there was calls to several binaries.
For one zone I had external calls to (in order):
echo, mkdir, printf(?), flock, ls, rm, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, mkdir, mkdir, mkdir, mkdir, install, install, chmod, chmod, chown, echo.
Some of these could be internal builtins, but the consumed time suggested otherwise.
Either way. Each install took about 26 ms and I could account the majority of that time in forking external programs and waiting for them to return.
So my conclusion is: The whole thing is a bit slow and bash doesn't help.
Regards,
Christian
>>
>> Thanks for any feedback.
>> BR,
>> Christian
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ptxdist] Speed up targetinstall of certain packages
2022-11-08 10:13 ` Christian Melki
@ 2022-11-08 12:10 ` Christian Melki
2022-11-11 16:08 ` Michael Olbrich
2022-11-08 12:50 ` Alexander Dahl
1 sibling, 1 reply; 8+ messages in thread
From: Christian Melki @ 2022-11-08 12:10 UTC (permalink / raw)
To: post, CG, Michael Olbrich; +Cc: ptxdist
So I did an experiment with bash loadables (builtins).
Turns out bash has a bunch of builtin examples.
I had ptxdist build me a bash-5.2 for x86_64.
With the "loadables" make target.
$ diff -urN ../lib/ptxdist-2022.11.0/rules/bash.make rules/bash.make
--- ../lib/ptxdist-2022.11.0/rules/bash.make 2022-10-21 16:54:50.000000000 +0200
+++ rules/bash.make 2022-11-08 12:51:38.609794625 +0100
@@ -77,6 +77,16 @@
--$(call ptx/wwo, PTXCONF_BASH_CURSES)-curses
# ----------------------------------------------------------------------------
+# Compile
+# ----------------------------------------------------------------------------
+
+$(STATEDIR)/bash.compile:
+ @$(call targetinfo)
+ @$(call world/compile, BASH)
+ @$(call compile, BASH, loadables)
+ @$(call touch)
+
+# ----------------------------------------------------------------------------
Since mkdir is slightly less than half of the external binary calls I had it replaced.
I copied the scripts/lib/ptxd_make_xpkg_pkg.sh to my build path
and created a shell-loadables directory there.
$ cp platform-secplatform-x86_64/build-target/bash-5.2/examples/loadables/mkdir scripts/lib/shell-loadables/
$ ls -1 scripts/lib/shell-loadables/
mkdir
$ diff -urN ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh scripts/lib/ptxd_make_xpkg_pkg.sh
--- ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh 2022-10-21 16:54:50.000000000 +0200
+++ scripts/lib/ptxd_make_xpkg_pkg.sh 2022-11-08 13:08:18.074550829 +0100
@@ -682,6 +682,8 @@
ptxd_install_file() {
local cmd="file"
+ #export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
+ enable -f scripts/lib/shell-loadables/mkdir mkdir
ptxd_install_file_impl "$@" ||
ptxd_install_error "install_file failed!"
}
without mkdir builtin:
finished target timezone.targetinstall.post
real 0m17,043s
user 0m13,522s
sys 0m4,138s
with mkdir builtin:
finished target timezone.targetinstall.post
real 0m12,345s
user 0m9,899s
sys 0m2,916s
On 11/8/22 11:13 AM, Christian Melki wrote:
>
>
> On 11/4/22 8:12 PM, Alexander Dahl wrote:
>> Hello Christian,
>>
>> Am Fri, Nov 04, 2022 at 03:37:02PM +0000 schrieb Gieseler, Christian:
>>> Hello,
>>>
>>> I have question regarding the speedup of daily work.
>>>
>>> We have frontend and backend of our webgui deployed with separate packages. Only task of these package is to deploy the files with
>>>
>>> @$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/, /var/www/,no)
>>>
>>> Compile and install stages are empty. The just call targetinfo and touch to skip the stages.
>>>
>>> The frontend depends on the backend and the backend obviously depends on our application which is called by the backend.
>>> So our web-frontend.in file looks like this:
>>> ## SECTION=project_specific
>>>
>>> config WEB_FRONTEND
>>> bool
>>> select APP_LAYER
>>> select WEB_BACKEND
>>> prompt "e-mode Web Frontend"
>>> help
>>>
>>> As expected if i clean and compile APP_LAYER the targetinstallstage of Backend and Frontend are executed again. However this is only a Run-Time only dependency. It is a third-party archive and install_tree takes quite some time even on fast build hosts. Even it if is just a minute it is annoying to spend the time waiting during image creation. Trying to solve that i found "if RUNTIME" für Run-Time only Dependencys in the documentation here:
>>>
>>> https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail
>>>
>>> So my expectation would be that if i change the webfrontend.in file like this:
>>>
>>> config WEB_FRONTEND
>>> bool
>>> select APP_LAYER if RUNTIME
>>> select WEB_BACKEND if RUNTIME
>>> prompt "e-mode Web Frontend"
>>> help
>>
>> That sounds reasonable and I would have done it the same.
>>
>>> The "if RUNTIME" would make sure that the targetinstall stage is not executed again if i just execute a "ptxdist clean app-layer" followed by a "ptxdist images". Same with ptxdist clean root; ptxdist images. It is clear that all targetinstall stages are executed again, but i would expect that the web-frontend is deployed earlier if no build dependency is given.
>>>
>>> Am i missing something, oder is the "if RUNTIME" Switch not working properly in my ptxdist-2018.12 version? Or does it have no effect on targetinstall stages?
>>
>> Not sure how that should behave. However if you want to speed up the
>> build: make sure you call ptxdist with -q or --quiet parameter. The
>> output on screen takes suprisingly much time, even with modern
>> terminals, and especially when doing targetinstall of many many files
>> (as usually the case with web frontends. been there, done that.)
>>
>> Greets
>> Alex
>>
>
> I have a slight disagreement here. I don't think the console is slow.
> So I did some investigation, mostly since the slowness bugs me too.
> I did a:
> export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
> as time measurement and trace in the shellscript in question (mostly scripts/lib/ptxd_make_xpkg_pkg.sh).
> This pretty linear progressive cpu consumption, albeit the big chunks were due to calling of external binaries.
>
> Then I did a timezone package clean and targetinstall.
> When targetinstalling a zone there was calls to several binaries.
>
> For one zone I had external calls to (in order):
> echo, mkdir, printf(?), flock, ls, rm, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, mkdir, mkdir, mkdir, mkdir, install, install, chmod, chmod, chown, echo.
>
> Some of these could be internal builtins, but the consumed time suggested otherwise.
> Either way. Each install took about 26 ms and I could account the majority of that time in forking external programs and waiting for them to return.
>
> So my conclusion is: The whole thing is a bit slow and bash doesn't help.
>
> Regards,
> Christian
>
>>>
>>> Thanks for any feedback.
>>> BR,
>>> Christian
>>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ptxdist] Speed up targetinstall of certain packages
2022-11-08 10:13 ` Christian Melki
2022-11-08 12:10 ` Christian Melki
@ 2022-11-08 12:50 ` Alexander Dahl
1 sibling, 0 replies; 8+ messages in thread
From: Alexander Dahl @ 2022-11-08 12:50 UTC (permalink / raw)
To: Christian Melki; +Cc: Michael Olbrich, post, ptxdist, CG
Hello everyone,
Am Tue, Nov 08, 2022 at 11:13:44AM +0100 schrieb Christian Melki:
>
>
> On 11/4/22 8:12 PM, Alexander Dahl wrote:
> > Not sure how that should behave. However if you want to speed up the
> > build: make sure you call ptxdist with -q or --quiet parameter. The
> > output on screen takes suprisingly much time, even with modern
> > terminals, and especially when doing targetinstall of many many files
> > (as usually the case with web frontends. been there, done that.)
> >
> > Greets
> > Alex
> >
>
> I have a slight disagreement here. I don't think the console is slow.
You're right. As already discussed in IRC yesterday: I did some
benchmarks yesterday on the targetinstall stage of a package using
install_tree to install a folder containing 900+ files. Turns out: no
significant difference between running with or without --quiet.
Greets
Alex
> So I did some investigation, mostly since the slowness bugs me too.
> I did a:
> export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
> as time measurement and trace in the shellscript in question (mostly scripts/lib/ptxd_make_xpkg_pkg.sh).
> This pretty linear progressive cpu consumption, albeit the big chunks were due to calling of external binaries.
>
> Then I did a timezone package clean and targetinstall.
> When targetinstalling a zone there was calls to several binaries.
>
> For one zone I had external calls to (in order):
> echo, mkdir, printf(?), flock, ls, rm, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, mkdir, mkdir, mkdir, mkdir, install, install, chmod, chmod, chown, echo.
>
> Some of these could be internal builtins, but the consumed time suggested otherwise.
> Either way. Each install took about 26 ms and I could account the majority of that time in forking external programs and waiting for them to return.
>
> So my conclusion is: The whole thing is a bit slow and bash doesn't help.
>
> Regards,
> Christian
>
> > >
> > > Thanks for any feedback.
> > > BR,
> > > Christian
> >
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ptxdist] Speed up targetinstall of certain packages
2022-11-08 12:10 ` Christian Melki
@ 2022-11-11 16:08 ` Michael Olbrich
2022-11-11 17:52 ` Christian Melki
0 siblings, 1 reply; 8+ messages in thread
From: Michael Olbrich @ 2022-11-11 16:08 UTC (permalink / raw)
To: Christian Melki; +Cc: post, ptxdist, CG
Hi,
On Tue, Nov 08, 2022 at 01:10:32PM +0100, Christian Melki wrote:
> So I did an experiment with bash loadables (builtins).
> Turns out bash has a bunch of builtin examples.
>
> I had ptxdist build me a bash-5.2 for x86_64.
> With the "loadables" make target.
>
> $ diff -urN ../lib/ptxdist-2022.11.0/rules/bash.make rules/bash.make
> --- ../lib/ptxdist-2022.11.0/rules/bash.make 2022-10-21 16:54:50.000000000 +0200
> +++ rules/bash.make 2022-11-08 12:51:38.609794625 +0100
> @@ -77,6 +77,16 @@
> --$(call ptx/wwo, PTXCONF_BASH_CURSES)-curses
> # ----------------------------------------------------------------------------
> +# Compile
> +# ----------------------------------------------------------------------------
> +
> +$(STATEDIR)/bash.compile:
> + @$(call targetinfo)
> + @$(call world/compile, BASH)
> + @$(call compile, BASH, loadables)
> + @$(call touch)
> +
> +# ----------------------------------------------------------------------------
>
> Since mkdir is slightly less than half of the external binary calls I had it replaced.
> I copied the scripts/lib/ptxd_make_xpkg_pkg.sh to my build path
> and created a shell-loadables directory there.
>
> $ cp platform-secplatform-x86_64/build-target/bash-5.2/examples/loadables/mkdir scripts/lib/shell-loadables/
>
> $ ls -1 scripts/lib/shell-loadables/
> mkdir
>
> $ diff -urN ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh scripts/lib/ptxd_make_xpkg_pkg.sh
> --- ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh 2022-10-21 16:54:50.000000000 +0200
> +++ scripts/lib/ptxd_make_xpkg_pkg.sh 2022-11-08 13:08:18.074550829 +0100
> @@ -682,6 +682,8 @@
> ptxd_install_file() {
> local cmd="file"
> + #export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
> + enable -f scripts/lib/shell-loadables/mkdir mkdir
> ptxd_install_file_impl "$@" ||
> ptxd_install_error "install_file failed!"
> }
>
> without mkdir builtin:
> finished target timezone.targetinstall.post
>
> real 0m17,043s
> user 0m13,522s
> sys 0m4,138s
>
> with mkdir builtin:
> finished target timezone.targetinstall.post
>
> real 0m12,345s
> user 0m9,899s
> sys 0m2,916s
So loadable builtins in bash is interesting, but not something we can rely
on at this time. But this whole thing got me thinking. Most mkdirs are
don't actually do anything. They are just in case the directory does not
exist yet. So we can do that:
if [ ! -d "${dir}" ]; then
mkdir -p "${dir}"
fi
That avoid most of the mkdirs and the speedup is considerably. Similar to
the builtin I think. And I found some more things to improve:
The 'flock' is only needed when building packages in parallel, so we can
skip it for things like "ptxdist drop foo.compile; ptxdist targetinstall
foo", and that's where the slow targetinstall hurts the moset.
Do the chmod/chown with the 'install' when possible.
And some stuff to improve the ptxdist startup time.
For "time ptxdist targetinstall timezone" I get:
Before:
real 0m29.167s
user 0m19.389s
sys 0m9.903s
After:
real 0m8.887s
user 0m5.954s
sys 0m2.470s
This stuff should hit master pretty soon.
Thanks for the inspiration :-).
Michael
> On 11/8/22 11:13 AM, Christian Melki wrote:
> >
> >
> > On 11/4/22 8:12 PM, Alexander Dahl wrote:
> > > Hello Christian,
> > >
> > > Am Fri, Nov 04, 2022 at 03:37:02PM +0000 schrieb Gieseler, Christian:
> > > > Hello,
> > > >
> > > > I have question regarding the speedup of daily work.
> > > >
> > > > We have frontend and backend of our webgui deployed with separate packages. Only task of these package is to deploy the files with
> > > >
> > > > @$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/, /var/www/,no)
> > > >
> > > > Compile and install stages are empty. The just call targetinfo and touch to skip the stages.
> > > >
> > > > The frontend depends on the backend and the backend obviously depends on our application which is called by the backend.
> > > > So our web-frontend.in file looks like this:
> > > > ## SECTION=project_specific
> > > >
> > > > config WEB_FRONTEND
> > > > bool
> > > > select APP_LAYER
> > > > select WEB_BACKEND
> > > > prompt "e-mode Web Frontend"
> > > > help
> > > >
> > > > As expected if i clean and compile APP_LAYER the targetinstallstage of Backend and Frontend are executed again. However this is only a Run-Time only dependency. It is a third-party archive and install_tree takes quite some time even on fast build hosts. Even it if is just a minute it is annoying to spend the time waiting during image creation. Trying to solve that i found "if RUNTIME" für Run-Time only Dependencys in the documentation here:
> > > >
> > > > https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail
> > > >
> > > > So my expectation would be that if i change the webfrontend.in file like this:
> > > >
> > > > config WEB_FRONTEND
> > > > bool
> > > > select APP_LAYER if RUNTIME
> > > > select WEB_BACKEND if RUNTIME
> > > > prompt "e-mode Web Frontend"
> > > > help
> > >
> > > That sounds reasonable and I would have done it the same.
> > >
> > > > The "if RUNTIME" would make sure that the targetinstall stage is not executed again if i just execute a "ptxdist clean app-layer" followed by a "ptxdist images". Same with ptxdist clean root; ptxdist images. It is clear that all targetinstall stages are executed again, but i would expect that the web-frontend is deployed earlier if no build dependency is given.
> > > >
> > > > Am i missing something, oder is the "if RUNTIME" Switch not working properly in my ptxdist-2018.12 version? Or does it have no effect on targetinstall stages?
> > >
> > > Not sure how that should behave. However if you want to speed up the
> > > build: make sure you call ptxdist with -q or --quiet parameter. The
> > > output on screen takes suprisingly much time, even with modern
> > > terminals, and especially when doing targetinstall of many many files
> > > (as usually the case with web frontends. been there, done that.)
> > >
> > > Greets
> > > Alex
> > >
> >
> > I have a slight disagreement here. I don't think the console is slow.
> > So I did some investigation, mostly since the slowness bugs me too.
> > I did a:
> > export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
> > as time measurement and trace in the shellscript in question (mostly scripts/lib/ptxd_make_xpkg_pkg.sh).
> > This pretty linear progressive cpu consumption, albeit the big chunks were due to calling of external binaries.
> >
> > Then I did a timezone package clean and targetinstall.
> > When targetinstalling a zone there was calls to several binaries.
> >
> > For one zone I had external calls to (in order):
> > echo, mkdir, printf(?), flock, ls, rm, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, mkdir, mkdir, mkdir, mkdir, install, install, chmod, chmod, chown, echo.
> >
> > Some of these could be internal builtins, but the consumed time suggested otherwise.
> > Either way. Each install took about 26 ms and I could account the majority of that time in forking external programs and waiting for them to return.
> >
> > So my conclusion is: The whole thing is a bit slow and bash doesn't help.
> >
> > Regards,
> > Christian
> >
> > > >
> > > > Thanks for any feedback.
> > > > BR,
> > > > Christian
> > >
> >
>
>
--
Pengutronix e.K. | |
Steuerwalder Str. 21 | http://www.pengutronix.de/ |
31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ptxdist] Speed up targetinstall of certain packages
2022-11-11 16:08 ` Michael Olbrich
@ 2022-11-11 17:52 ` Christian Melki
0 siblings, 0 replies; 8+ messages in thread
From: Christian Melki @ 2022-11-11 17:52 UTC (permalink / raw)
To: Michael Olbrich; +Cc: post, ptxdist, CG
On 11/11/22 17:08, Michael Olbrich wrote:
> Hi,
>
> On Tue, Nov 08, 2022 at 01:10:32PM +0100, Christian Melki wrote:
>> So I did an experiment with bash loadables (builtins).
>> Turns out bash has a bunch of builtin examples.
>>
>> I had ptxdist build me a bash-5.2 for x86_64.
>> With the "loadables" make target.
>>
>> $ diff -urN ../lib/ptxdist-2022.11.0/rules/bash.make rules/bash.make
>> --- ../lib/ptxdist-2022.11.0/rules/bash.make 2022-10-21 16:54:50.000000000 +0200
>> +++ rules/bash.make 2022-11-08 12:51:38.609794625 +0100
>> @@ -77,6 +77,16 @@
>> --$(call ptx/wwo, PTXCONF_BASH_CURSES)-curses
>> # ----------------------------------------------------------------------------
>> +# Compile
>> +# ----------------------------------------------------------------------------
>> +
>> +$(STATEDIR)/bash.compile:
>> + @$(call targetinfo)
>> + @$(call world/compile, BASH)
>> + @$(call compile, BASH, loadables)
>> + @$(call touch)
>> +
>> +# ----------------------------------------------------------------------------
>>
>> Since mkdir is slightly less than half of the external binary calls I had it replaced.
>> I copied the scripts/lib/ptxd_make_xpkg_pkg.sh to my build path
>> and created a shell-loadables directory there.
>>
>> $ cp platform-secplatform-x86_64/build-target/bash-5.2/examples/loadables/mkdir scripts/lib/shell-loadables/
>>
>> $ ls -1 scripts/lib/shell-loadables/
>> mkdir
>>
>> $ diff -urN ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh scripts/lib/ptxd_make_xpkg_pkg.sh
>> --- ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh 2022-10-21 16:54:50.000000000 +0200
>> +++ scripts/lib/ptxd_make_xpkg_pkg.sh 2022-11-08 13:08:18.074550829 +0100
>> @@ -682,6 +682,8 @@
>> ptxd_install_file() {
>> local cmd="file"
>> + #export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
>> + enable -f scripts/lib/shell-loadables/mkdir mkdir
>> ptxd_install_file_impl "$@" ||
>> ptxd_install_error "install_file failed!"
>> }
>>
>> without mkdir builtin:
>> finished target timezone.targetinstall.post
>>
>> real 0m17,043s
>> user 0m13,522s
>> sys 0m4,138s
>>
>> with mkdir builtin:
>> finished target timezone.targetinstall.post
>>
>> real 0m12,345s
>> user 0m9,899s
>> sys 0m2,916s
>
> So loadable builtins in bash is interesting, but not something we can rely
> on at this time. But this whole thing got me thinking. Most mkdirs are
> don't actually do anything. They are just in case the directory does not
> exist yet. So we can do that:
>
> if [ ! -d "${dir}" ]; then
> mkdir -p "${dir}"
> fi
>
> That avoid most of the mkdirs and the speedup is considerably. Similar to
> the builtin I think. And I found some more things to improve:
>
> The 'flock' is only needed when building packages in parallel, so we can
> skip it for things like "ptxdist drop foo.compile; ptxdist targetinstall
> foo", and that's where the slow targetinstall hurts the moset.
>
> Do the chmod/chown with the 'install' when possible.
>
> And some stuff to improve the ptxdist startup time.
>
> For "time ptxdist targetinstall timezone" I get:
>
> Before:
> real 0m29.167s
> user 0m19.389s
> sys 0m9.903s
>
> After:
> real 0m8.887s
> user 0m5.954s
> sys 0m2.470s
>
> This stuff should hit master pretty soon.
>
> Thanks for the inspiration :-).
>
> Michael
>
Super!
The end goal was reached either way.
A faster ptxdist! Wee! :D
And thanks for the kind words,
Christian
>> On 11/8/22 11:13 AM, Christian Melki wrote:
>>>
>>>
>>> On 11/4/22 8:12 PM, Alexander Dahl wrote:
>>>> Hello Christian,
>>>>
>>>> Am Fri, Nov 04, 2022 at 03:37:02PM +0000 schrieb Gieseler, Christian:
>>>>> Hello,
>>>>>
>>>>> I have question regarding the speedup of daily work.
>>>>>
>>>>> We have frontend and backend of our webgui deployed with separate packages. Only task of these package is to deploy the files with
>>>>>
>>>>> @$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/, /var/www/,no)
>>>>>
>>>>> Compile and install stages are empty. The just call targetinfo and touch to skip the stages.
>>>>>
>>>>> The frontend depends on the backend and the backend obviously depends on our application which is called by the backend.
>>>>> So our web-frontend.in file looks like this:
>>>>> ## SECTION=project_specific
>>>>>
>>>>> config WEB_FRONTEND
>>>>> bool
>>>>> select APP_LAYER
>>>>> select WEB_BACKEND
>>>>> prompt "e-mode Web Frontend"
>>>>> help
>>>>>
>>>>> As expected if i clean and compile APP_LAYER the targetinstallstage of Backend and Frontend are executed again. However this is only a Run-Time only dependency. It is a third-party archive and install_tree takes quite some time even on fast build hosts. Even it if is just a minute it is annoying to spend the time waiting during image creation. Trying to solve that i found "if RUNTIME" für Run-Time only Dependencys in the documentation here:
>>>>>
>>>>> https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail
>>>>>
>>>>> So my expectation would be that if i change the webfrontend.in file like this:
>>>>>
>>>>> config WEB_FRONTEND
>>>>> bool
>>>>> select APP_LAYER if RUNTIME
>>>>> select WEB_BACKEND if RUNTIME
>>>>> prompt "e-mode Web Frontend"
>>>>> help
>>>>
>>>> That sounds reasonable and I would have done it the same.
>>>>
>>>>> The "if RUNTIME" would make sure that the targetinstall stage is not executed again if i just execute a "ptxdist clean app-layer" followed by a "ptxdist images". Same with ptxdist clean root; ptxdist images. It is clear that all targetinstall stages are executed again, but i would expect that the web-frontend is deployed earlier if no build dependency is given.
>>>>>
>>>>> Am i missing something, oder is the "if RUNTIME" Switch not working properly in my ptxdist-2018.12 version? Or does it have no effect on targetinstall stages?
>>>>
>>>> Not sure how that should behave. However if you want to speed up the
>>>> build: make sure you call ptxdist with -q or --quiet parameter. The
>>>> output on screen takes suprisingly much time, even with modern
>>>> terminals, and especially when doing targetinstall of many many files
>>>> (as usually the case with web frontends. been there, done that.)
>>>>
>>>> Greets
>>>> Alex
>>>>
>>>
>>> I have a slight disagreement here. I don't think the console is slow.
>>> So I did some investigation, mostly since the slowness bugs me too.
>>> I did a:
>>> export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x;
>>> as time measurement and trace in the shellscript in question (mostly scripts/lib/ptxd_make_xpkg_pkg.sh).
>>> This pretty linear progressive cpu consumption, albeit the big chunks were due to calling of external binaries.
>>>
>>> Then I did a timezone package clean and targetinstall.
>>> When targetinstalling a zone there was calls to several binaries.
>>>
>>> For one zone I had external calls to (in order):
>>> echo, mkdir, printf(?), flock, ls, rm, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, mkdir, mkdir, mkdir, mkdir, install, install, chmod, chmod, chown, echo.
>>>
>>> Some of these could be internal builtins, but the consumed time suggested otherwise.
>>> Either way. Each install took about 26 ms and I could account the majority of that time in forking external programs and waiting for them to return.
>>>
>>> So my conclusion is: The whole thing is a bit slow and bash doesn't help.
>>>
>>> Regards,
>>> Christian
>>>
>>>>>
>>>>> Thanks for any feedback.
>>>>> BR,
>>>>> Christian
>>>>
>>>
>>
>>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2022-11-11 17:55 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-04 15:37 [ptxdist] Speed up targetinstall of certain packages Gieseler, Christian
2022-11-04 19:12 ` Alexander Dahl
2022-11-08 10:13 ` Christian Melki
2022-11-08 12:10 ` Christian Melki
2022-11-11 16:08 ` Michael Olbrich
2022-11-11 17:52 ` Christian Melki
2022-11-08 12:50 ` Alexander Dahl
2022-11-08 8:25 ` Michael Olbrich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox