From: Roland Hieber <rhi@pengutronix.de>
To: ptxdist@pengutronix.de
Cc: Roland Hieber <rhi@pengutronix.de>
Subject: [ptxdist] [PATCH 2/2] ptxd_lib_dgen: optimize ordering only when packages are being fully built
Date: Mon, 25 May 2020 11:28:20 +0200 [thread overview]
Message-ID: <20200525092820.818-2-rhi@pengutronix.de> (raw)
In-Reply-To: <20200525092820.818-1-rhi@pengutronix.de>
There are use cases where -j is useful to have on the command line to
speed up runs with large amount of packages, but when targetinstall
stages are not needed at all. For example:
ptxdist -j license-check
ptxdist -j make license-report
ptxdist -j extract <long list of package names>
Commit 2e695987211c2c2e6e19 ("ptxd_lib_dgen: add order-only dependency
to extract") introduced better interleaving of extract stages to
optimise IO load during full builds. This also had the effect that in
the above cases all targetinstall stages of dependencies were built too,
even when they are not needed at all. Improve that use case by adding a
flag in the environment and only set it in case of 'ptxdist go',
'ptxdist images' or 'ptxdist image <image-name>'.
Signed-off-by: Roland Hieber <rhi@pengutronix.de>
---
bin/ptxdist | 3 +++
scripts/lib/ptxd_lib_dgen.awk | 5 +++--
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/bin/ptxdist b/bin/ptxdist
index 4a372ff6392c..6034d170fcdc 100755
--- a/bin/ptxdist
+++ b/bin/ptxdist
@@ -1764,10 +1764,12 @@ EOF
exit
;;
go|images)
+ PTXDIST_OPTIMIZE_IO=true
do_${cmd}
exit
;;
image)
+ PTXDIST_OPTIMIZE_IO=true
if [ ${#} -eq 0 ]; then
echo "No image given."
exit 1
@@ -2564,6 +2566,7 @@ setup_export() {
PTXDIST_ICECC_CREATE_ENV \
PTXDIST_LOG_PROMPT \
PTXDIST_OUTPUT_SYNC \
+ PTXDIST_OPTIMIZE_IO \
PTXDIST_DIRTY \
PTXDIST_FORCE \
PTXDIST_PEDANTIC \
diff --git a/scripts/lib/ptxd_lib_dgen.awk b/scripts/lib/ptxd_lib_dgen.awk
index 8ee88827b1f8..8c09c531129e 100644
--- a/scripts/lib/ptxd_lib_dgen.awk
+++ b/scripts/lib/ptxd_lib_dgen.awk
@@ -17,6 +17,7 @@ BEGIN {
DGEN_DEPS_PRE = ENVIRON["PTX_DGEN_DEPS_PRE"];
DGEN_DEPS_POST = ENVIRON["PTX_DGEN_DEPS_POST"];
DGEN_RULESFILES_MAKE = ENVIRON["PTX_DGEN_RULESFILES_MAKE"];
+ OPTIMIZE_IO = ENVIRON["PTXDIST_OPTIMIZE_IO"]
PTXDIST_TEMPDIR = ENVIRON["PTXDIST_TEMPDIR"];
PARALLEL = ENVIRON["PTXDIST_PARALLELMFLAGS_EXTERN"]
DIRTY = ENVIRON["PTXDIST_DIRTY"];
@@ -503,7 +504,7 @@ function write_deps_pkg_active(this_PKG, this_pkg, prefix) {
for (i = 1; i <= n; i++) {
this_dep = PKG_to_pkg[this_DEP_array[i]]
- if (PARALLEL != "-j1")
+ if (PARALLEL != "-j1" && OPTIMIZE_IO == "true")
print "$(STATEDIR)/" this_pkg ".extract:| " "$(STATEDIR)/" this_dep ".install.post" > DGEN_DEPS_POST;
print "$(STATEDIR)/" this_pkg ".extract.post:" DEP " $(STATEDIR)/" this_dep ".install.post" > DGEN_DEPS_POST;
print "$(STATEDIR)/" this_pkg ".install.unpack:" DEP " $(STATEDIR)/" this_dep ".install.post" > DGEN_DEPS_POST;
@@ -545,7 +546,7 @@ function write_deps_pkg_active_virtual(this_PKG, this_pkg, prefix) {
else
virtual = "base";
}
- if (PARALLEL != "-j1")
+ if (PARALLEL != "-j1" && OPTIMIZE_IO == "true")
print "$(STATEDIR)/" this_pkg ".extract:| " "$(STATEDIR)/" virtual ".install" > DGEN_DEPS_POST;
print "$(STATEDIR)/" this_pkg ".extract.post:" DEP " $(STATEDIR)/" virtual ".install" > DGEN_DEPS_POST;
print "$(STATEDIR)/" this_pkg ".install.unpack:" DEP " $(STATEDIR)/" virtual ".install" > DGEN_DEPS_POST;
--
2.26.2
_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
next prev parent reply other threads:[~2020-05-25 9:28 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-25 9:28 [ptxdist] [PATCH 1/2] ptxdist: add licensecheck command Roland Hieber
2020-05-25 9:28 ` Roland Hieber [this message]
2020-05-25 11:46 ` [ptxdist] [PATCH 2/2] ptxd_lib_dgen: optimize ordering only when packages are being fully built Roland Hieber
2020-05-25 12:47 ` Michael Olbrich
2020-05-25 20:22 ` [ptxdist] [PATCH v2 2/2] ptxd_lib_dgen: optimize ordering only when dependencies are needed Roland Hieber
2020-06-05 5:59 ` [ptxdist] [APPLIED] " Michael Olbrich
2020-05-29 5:59 ` [ptxdist] [PATCH 1/2] ptxdist: add licensecheck command Michael Olbrich
2020-05-29 9:13 ` [ptxdist] [PATCH v2 " Roland Hieber
2020-06-05 5:59 ` [ptxdist] [APPLIED] " Michael Olbrich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200525092820.818-2-rhi@pengutronix.de \
--to=rhi@pengutronix.de \
--cc=ptxdist@pengutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox