* [ptxdist] [PATCH] image-root-cpio: Add compressor options to the env. @ 2022-03-03 22:25 Christian Melki 2022-03-04 9:09 ` Michael Olbrich 0 siblings, 1 reply; 4+ messages in thread From: Christian Melki @ 2022-03-03 22:25 UTC (permalink / raw) To: ptxdist * xz needs crc32 instead of crc64 for the kernel cpio. * Set zstd compression level to the same as for the kernel (19). * Use multithreading for zstd (deterministic, afaiu). Actively decided against -T0 for xz, as compression is not deterministic when number of threads varies. Signed-off-by: Christian Melki <christian.melki@t2data.com> --- rules/image-root-cpio.make | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rules/image-root-cpio.make b/rules/image-root-cpio.make index fdb0ccfff..eb09dd53e 100644 --- a/rules/image-root-cpio.make +++ b/rules/image-root-cpio.make @@ -26,6 +26,9 @@ IMAGE_ROOT_CPIO_CONFIG := cpio.config IMAGE_ROOT_CPIO_ENV := \ FORMAT="newc" \ + ZSTD_CLEVEL="19" \ + ZSTD_NBTHREADS="0" \ + XZ_OPT="--check=crc32" \ COMPRESS=$(call remove_quotes, $(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL)) ifdef PTXCONF_IMAGE_ROOT_CPIO -- 2.30.2 _______________________________________________ ptxdist mailing list ptxdist@pengutronix.de To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [ptxdist] [PATCH] image-root-cpio: Add compressor options to the env. 2022-03-03 22:25 [ptxdist] [PATCH] image-root-cpio: Add compressor options to the env Christian Melki @ 2022-03-04 9:09 ` Michael Olbrich 2022-03-04 9:21 ` Christian Melki 0 siblings, 1 reply; 4+ messages in thread From: Michael Olbrich @ 2022-03-04 9:09 UTC (permalink / raw) To: Christian Melki; +Cc: ptxdist On Thu, Mar 03, 2022 at 11:25:05PM +0100, Christian Melki wrote: > * xz needs crc32 instead of crc64 for the kernel cpio. Good point. I didn't know about that. > * Set zstd compression level to the same as for the kernel (19). Makes sense. > * Use multithreading for zstd (deterministic, afaiu). It is. I looked into this because I was curious. And it makes sense. zstd was developed at a time when reproducibility was already important. > Actively decided against -T0 for xz, as compression > is not deterministic when number of threads varies. We definitively want to be reproducible here. > Signed-off-by: Christian Melki <christian.melki@t2data.com> > --- > rules/image-root-cpio.make | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/rules/image-root-cpio.make b/rules/image-root-cpio.make > index fdb0ccfff..eb09dd53e 100644 > --- a/rules/image-root-cpio.make > +++ b/rules/image-root-cpio.make > @@ -26,6 +26,9 @@ IMAGE_ROOT_CPIO_CONFIG := cpio.config > > IMAGE_ROOT_CPIO_ENV := \ > FORMAT="newc" \ > + ZSTD_CLEVEL="19" \ > + ZSTD_NBTHREADS="0" \ > + XZ_OPT="--check=crc32" \ > COMPRESS=$(call remove_quotes, $(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL)) Add the options to the PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL defaults and keep the quotes here: COMPRESS=$(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL) genimage explicitly does not quote to make to possible to specify the compressor with options. At least I thinks that should work. And maybe we could add a IMAGE_ROOT_CPIO_COMPRESSION_MODE_CUSTOM and then add 'prompt "suffix" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_CUSTOM' to the suffix and similar for the util. This way the user could tweak things if absolutely necessary. What do you think? Michael > > ifdef PTXCONF_IMAGE_ROOT_CPIO > -- > 2.30.2 > > > _______________________________________________ > ptxdist mailing list > ptxdist@pengutronix.de > To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de > -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | _______________________________________________ ptxdist mailing list ptxdist@pengutronix.de To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [ptxdist] [PATCH] image-root-cpio: Add compressor options to the env. 2022-03-04 9:09 ` Michael Olbrich @ 2022-03-04 9:21 ` Christian Melki 2022-03-04 10:48 ` Michael Olbrich 0 siblings, 1 reply; 4+ messages in thread From: Christian Melki @ 2022-03-04 9:21 UTC (permalink / raw) To: Michael Olbrich; +Cc: ptxdist On 3/4/22 10:09 AM, Michael Olbrich wrote: > On Thu, Mar 03, 2022 at 11:25:05PM +0100, Christian Melki wrote: >> * xz needs crc32 instead of crc64 for the kernel cpio. > > Good point. I didn't know about that. > >> * Set zstd compression level to the same as for the kernel (19). > > Makes sense. > >> * Use multithreading for zstd (deterministic, afaiu). > > It is. I looked into this because I was curious. And it makes sense. zstd > was developed at a time when reproducibility was already important. > >> Actively decided against -T0 for xz, as compression >> is not deterministic when number of threads varies. > > We definitively want to be reproducible here. > >> Signed-off-by: Christian Melki <christian.melki@t2data.com> >> --- >> rules/image-root-cpio.make | 3 +++ >> 1 file changed, 3 insertions(+) >> >> diff --git a/rules/image-root-cpio.make b/rules/image-root-cpio.make >> index fdb0ccfff..eb09dd53e 100644 >> --- a/rules/image-root-cpio.make >> +++ b/rules/image-root-cpio.make >> @@ -26,6 +26,9 @@ IMAGE_ROOT_CPIO_CONFIG := cpio.config >> >> IMAGE_ROOT_CPIO_ENV := \ >> FORMAT="newc" \ >> + ZSTD_CLEVEL="19" \ >> + ZSTD_NBTHREADS="0" \ >> + XZ_OPT="--check=crc32" \ >> COMPRESS=$(call remove_quotes, $(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL)) > > Add the options to the PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL defaults > and keep the quotes here: > > COMPRESS=$(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL) > > genimage explicitly does not quote to make to possible to specify the > compressor with options. At least I thinks that should work. > > And maybe we could add a IMAGE_ROOT_CPIO_COMPRESSION_MODE_CUSTOM and then > add 'prompt "suffix" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_CUSTOM' to the > suffix and similar for the util. > > This way the user could tweak things if absolutely necessary. > > What do you think? I don't mind, will fix. But custom compressors will be outside of the ptxdist dependency scope for building host binaries? I can make a note about it in the help? > > Michael > >> >> ifdef PTXCONF_IMAGE_ROOT_CPIO >> -- >> 2.30.2 >> >> >> _______________________________________________ >> ptxdist mailing list >> ptxdist@pengutronix.de >> To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de >> > _______________________________________________ ptxdist mailing list ptxdist@pengutronix.de To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [ptxdist] [PATCH] image-root-cpio: Add compressor options to the env. 2022-03-04 9:21 ` Christian Melki @ 2022-03-04 10:48 ` Michael Olbrich 0 siblings, 0 replies; 4+ messages in thread From: Michael Olbrich @ 2022-03-04 10:48 UTC (permalink / raw) To: Christian Melki; +Cc: ptxdist On Fri, Mar 04, 2022 at 10:21:24AM +0100, Christian Melki wrote: > > > On 3/4/22 10:09 AM, Michael Olbrich wrote: > > On Thu, Mar 03, 2022 at 11:25:05PM +0100, Christian Melki wrote: > > > * xz needs crc32 instead of crc64 for the kernel cpio. > > > > Good point. I didn't know about that. > > > > > * Set zstd compression level to the same as for the kernel (19). > > > > Makes sense. > > > > > * Use multithreading for zstd (deterministic, afaiu). > > > > It is. I looked into this because I was curious. And it makes sense. zstd > > was developed at a time when reproducibility was already important. > > > > > Actively decided against -T0 for xz, as compression > > > is not deterministic when number of threads varies. > > > > We definitively want to be reproducible here. > > > > > Signed-off-by: Christian Melki <christian.melki@t2data.com> > > > --- > > > rules/image-root-cpio.make | 3 +++ > > > 1 file changed, 3 insertions(+) > > > > > > diff --git a/rules/image-root-cpio.make b/rules/image-root-cpio.make > > > index fdb0ccfff..eb09dd53e 100644 > > > --- a/rules/image-root-cpio.make > > > +++ b/rules/image-root-cpio.make > > > @@ -26,6 +26,9 @@ IMAGE_ROOT_CPIO_CONFIG := cpio.config > > > IMAGE_ROOT_CPIO_ENV := \ > > > FORMAT="newc" \ > > > + ZSTD_CLEVEL="19" \ > > > + ZSTD_NBTHREADS="0" \ > > > + XZ_OPT="--check=crc32" \ > > > COMPRESS=$(call remove_quotes, $(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL)) > > > > Add the options to the PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL defaults > > and keep the quotes here: > > > > COMPRESS=$(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL) > > > > genimage explicitly does not quote to make to possible to specify the > > compressor with options. At least I thinks that should work. > > > > And maybe we could add a IMAGE_ROOT_CPIO_COMPRESSION_MODE_CUSTOM and then > > add 'prompt "suffix" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_CUSTOM' to the > > suffix and similar for the util. > > > > This way the user could tweak things if absolutely necessary. > > > > What do you think? > > I don't mind, will fix. But custom compressors will be outside of the > ptxdist dependency scope for building host binaries? > I can make a note about it in the help? Right, I forgot about that. That's not good. So maybe: config IMAGE_ROOT_CPIO_CUSTOM_COMPRESSION bool prompt "customise the compression command" outside the choice and only add the optional prompt to the util. So we still choose a type with its dependency. We could also split the options into a separat symbol and only add the prompt there, but I'm not so sure if that makes it better. Michael > > > > Michael > > > > > ifdef PTXCONF_IMAGE_ROOT_CPIO > > > -- > > > 2.30.2 > > > > > > > > > _______________________________________________ > > > ptxdist mailing list > > > ptxdist@pengutronix.de > > > To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de > > > > > > > _______________________________________________ > ptxdist mailing list > ptxdist@pengutronix.de > To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de > -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | _______________________________________________ ptxdist mailing list ptxdist@pengutronix.de To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-03-04 10:48 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-03-03 22:25 [ptxdist] [PATCH] image-root-cpio: Add compressor options to the env Christian Melki 2022-03-04 9:09 ` Michael Olbrich 2022-03-04 9:21 ` Christian Melki 2022-03-04 10:48 ` Michael Olbrich
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox