* [ptxdist] [PATCH] image-root-cpio: Add compressor options to the env.
@ 2022-03-03 22:25 Christian Melki
2022-03-04 9:09 ` Michael Olbrich
0 siblings, 1 reply; 4+ messages in thread
From: Christian Melki @ 2022-03-03 22:25 UTC (permalink / raw)
To: ptxdist
* xz needs crc32 instead of crc64 for the kernel cpio.
* Set zstd compression level to the same as for the kernel (19).
* Use multithreading for zstd (deterministic, afaiu).
Actively decided against -T0 for xz, as compression
is not deterministic when number of threads varies.
Signed-off-by: Christian Melki <christian.melki@t2data.com>
---
rules/image-root-cpio.make | 3 +++
1 file changed, 3 insertions(+)
diff --git a/rules/image-root-cpio.make b/rules/image-root-cpio.make
index fdb0ccfff..eb09dd53e 100644
--- a/rules/image-root-cpio.make
+++ b/rules/image-root-cpio.make
@@ -26,6 +26,9 @@ IMAGE_ROOT_CPIO_CONFIG := cpio.config
IMAGE_ROOT_CPIO_ENV := \
FORMAT="newc" \
+ ZSTD_CLEVEL="19" \
+ ZSTD_NBTHREADS="0" \
+ XZ_OPT="--check=crc32" \
COMPRESS=$(call remove_quotes, $(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL))
ifdef PTXCONF_IMAGE_ROOT_CPIO
--
2.30.2
_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [ptxdist] [PATCH] image-root-cpio: Add compressor options to the env.
2022-03-03 22:25 [ptxdist] [PATCH] image-root-cpio: Add compressor options to the env Christian Melki
@ 2022-03-04 9:09 ` Michael Olbrich
2022-03-04 9:21 ` Christian Melki
0 siblings, 1 reply; 4+ messages in thread
From: Michael Olbrich @ 2022-03-04 9:09 UTC (permalink / raw)
To: Christian Melki; +Cc: ptxdist
On Thu, Mar 03, 2022 at 11:25:05PM +0100, Christian Melki wrote:
> * xz needs crc32 instead of crc64 for the kernel cpio.
Good point. I didn't know about that.
> * Set zstd compression level to the same as for the kernel (19).
Makes sense.
> * Use multithreading for zstd (deterministic, afaiu).
It is. I looked into this because I was curious. And it makes sense. zstd
was developed at a time when reproducibility was already important.
> Actively decided against -T0 for xz, as compression
> is not deterministic when number of threads varies.
We definitively want to be reproducible here.
> Signed-off-by: Christian Melki <christian.melki@t2data.com>
> ---
> rules/image-root-cpio.make | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/rules/image-root-cpio.make b/rules/image-root-cpio.make
> index fdb0ccfff..eb09dd53e 100644
> --- a/rules/image-root-cpio.make
> +++ b/rules/image-root-cpio.make
> @@ -26,6 +26,9 @@ IMAGE_ROOT_CPIO_CONFIG := cpio.config
>
> IMAGE_ROOT_CPIO_ENV := \
> FORMAT="newc" \
> + ZSTD_CLEVEL="19" \
> + ZSTD_NBTHREADS="0" \
> + XZ_OPT="--check=crc32" \
> COMPRESS=$(call remove_quotes, $(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL))
Add the options to the PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL defaults
and keep the quotes here:
COMPRESS=$(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL)
genimage explicitly does not quote to make to possible to specify the
compressor with options. At least I thinks that should work.
And maybe we could add a IMAGE_ROOT_CPIO_COMPRESSION_MODE_CUSTOM and then
add 'prompt "suffix" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_CUSTOM' to the
suffix and similar for the util.
This way the user could tweak things if absolutely necessary.
What do you think?
Michael
>
> ifdef PTXCONF_IMAGE_ROOT_CPIO
> --
> 2.30.2
>
>
> _______________________________________________
> ptxdist mailing list
> ptxdist@pengutronix.de
> To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de
>
--
Pengutronix e.K. | |
Steuerwalder Str. 21 | http://www.pengutronix.de/ |
31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |
_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [ptxdist] [PATCH] image-root-cpio: Add compressor options to the env.
2022-03-04 9:09 ` Michael Olbrich
@ 2022-03-04 9:21 ` Christian Melki
2022-03-04 10:48 ` Michael Olbrich
0 siblings, 1 reply; 4+ messages in thread
From: Christian Melki @ 2022-03-04 9:21 UTC (permalink / raw)
To: Michael Olbrich; +Cc: ptxdist
On 3/4/22 10:09 AM, Michael Olbrich wrote:
> On Thu, Mar 03, 2022 at 11:25:05PM +0100, Christian Melki wrote:
>> * xz needs crc32 instead of crc64 for the kernel cpio.
>
> Good point. I didn't know about that.
>
>> * Set zstd compression level to the same as for the kernel (19).
>
> Makes sense.
>
>> * Use multithreading for zstd (deterministic, afaiu).
>
> It is. I looked into this because I was curious. And it makes sense. zstd
> was developed at a time when reproducibility was already important.
>
>> Actively decided against -T0 for xz, as compression
>> is not deterministic when number of threads varies.
>
> We definitively want to be reproducible here.
>
>> Signed-off-by: Christian Melki <christian.melki@t2data.com>
>> ---
>> rules/image-root-cpio.make | 3 +++
>> 1 file changed, 3 insertions(+)
>>
>> diff --git a/rules/image-root-cpio.make b/rules/image-root-cpio.make
>> index fdb0ccfff..eb09dd53e 100644
>> --- a/rules/image-root-cpio.make
>> +++ b/rules/image-root-cpio.make
>> @@ -26,6 +26,9 @@ IMAGE_ROOT_CPIO_CONFIG := cpio.config
>>
>> IMAGE_ROOT_CPIO_ENV := \
>> FORMAT="newc" \
>> + ZSTD_CLEVEL="19" \
>> + ZSTD_NBTHREADS="0" \
>> + XZ_OPT="--check=crc32" \
>> COMPRESS=$(call remove_quotes, $(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL))
>
> Add the options to the PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL defaults
> and keep the quotes here:
>
> COMPRESS=$(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL)
>
> genimage explicitly does not quote to make to possible to specify the
> compressor with options. At least I thinks that should work.
>
> And maybe we could add a IMAGE_ROOT_CPIO_COMPRESSION_MODE_CUSTOM and then
> add 'prompt "suffix" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_CUSTOM' to the
> suffix and similar for the util.
>
> This way the user could tweak things if absolutely necessary.
>
> What do you think?
I don't mind, will fix. But custom compressors will be outside of the
ptxdist dependency scope for building host binaries?
I can make a note about it in the help?
>
> Michael
>
>>
>> ifdef PTXCONF_IMAGE_ROOT_CPIO
>> --
>> 2.30.2
>>
>>
>> _______________________________________________
>> ptxdist mailing list
>> ptxdist@pengutronix.de
>> To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de
>>
>
_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [ptxdist] [PATCH] image-root-cpio: Add compressor options to the env.
2022-03-04 9:21 ` Christian Melki
@ 2022-03-04 10:48 ` Michael Olbrich
0 siblings, 0 replies; 4+ messages in thread
From: Michael Olbrich @ 2022-03-04 10:48 UTC (permalink / raw)
To: Christian Melki; +Cc: ptxdist
On Fri, Mar 04, 2022 at 10:21:24AM +0100, Christian Melki wrote:
>
>
> On 3/4/22 10:09 AM, Michael Olbrich wrote:
> > On Thu, Mar 03, 2022 at 11:25:05PM +0100, Christian Melki wrote:
> > > * xz needs crc32 instead of crc64 for the kernel cpio.
> >
> > Good point. I didn't know about that.
> >
> > > * Set zstd compression level to the same as for the kernel (19).
> >
> > Makes sense.
> >
> > > * Use multithreading for zstd (deterministic, afaiu).
> >
> > It is. I looked into this because I was curious. And it makes sense. zstd
> > was developed at a time when reproducibility was already important.
> >
> > > Actively decided against -T0 for xz, as compression
> > > is not deterministic when number of threads varies.
> >
> > We definitively want to be reproducible here.
> >
> > > Signed-off-by: Christian Melki <christian.melki@t2data.com>
> > > ---
> > > rules/image-root-cpio.make | 3 +++
> > > 1 file changed, 3 insertions(+)
> > >
> > > diff --git a/rules/image-root-cpio.make b/rules/image-root-cpio.make
> > > index fdb0ccfff..eb09dd53e 100644
> > > --- a/rules/image-root-cpio.make
> > > +++ b/rules/image-root-cpio.make
> > > @@ -26,6 +26,9 @@ IMAGE_ROOT_CPIO_CONFIG := cpio.config
> > > IMAGE_ROOT_CPIO_ENV := \
> > > FORMAT="newc" \
> > > + ZSTD_CLEVEL="19" \
> > > + ZSTD_NBTHREADS="0" \
> > > + XZ_OPT="--check=crc32" \
> > > COMPRESS=$(call remove_quotes, $(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL))
> >
> > Add the options to the PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL defaults
> > and keep the quotes here:
> >
> > COMPRESS=$(PTXCONF_IMAGE_ROOT_CPIO_COMPRESSION_UTIL)
> >
> > genimage explicitly does not quote to make to possible to specify the
> > compressor with options. At least I thinks that should work.
> >
> > And maybe we could add a IMAGE_ROOT_CPIO_COMPRESSION_MODE_CUSTOM and then
> > add 'prompt "suffix" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_CUSTOM' to the
> > suffix and similar for the util.
> >
> > This way the user could tweak things if absolutely necessary.
> >
> > What do you think?
>
> I don't mind, will fix. But custom compressors will be outside of the
> ptxdist dependency scope for building host binaries?
> I can make a note about it in the help?
Right, I forgot about that. That's not good. So maybe:
config IMAGE_ROOT_CPIO_CUSTOM_COMPRESSION
bool
prompt "customise the compression command"
outside the choice and only add the optional prompt to the util. So we
still choose a type with its dependency.
We could also split the options into a separat symbol and only add the
prompt there, but I'm not so sure if that makes it better.
Michael
> >
> > Michael
> >
> > > ifdef PTXCONF_IMAGE_ROOT_CPIO
> > > --
> > > 2.30.2
> > >
> > >
> > > _______________________________________________
> > > ptxdist mailing list
> > > ptxdist@pengutronix.de
> > > To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de
> > >
> >
>
> _______________________________________________
> ptxdist mailing list
> ptxdist@pengutronix.de
> To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de
>
--
Pengutronix e.K. | |
Steuerwalder Str. 21 | http://www.pengutronix.de/ |
31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |
_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-03-04 10:48 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-03 22:25 [ptxdist] [PATCH] image-root-cpio: Add compressor options to the env Christian Melki
2022-03-04 9:09 ` Michael Olbrich
2022-03-04 9:21 ` Christian Melki
2022-03-04 10:48 ` Michael Olbrich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox