* [ptxdist] [PATCH] image-root-cpio.in: Use multithreaded XZ (revisited).
@ 2024-02-25 14:49 Christian Melki
2024-03-07 16:04 ` Michael Olbrich
0 siblings, 1 reply; 3+ messages in thread
From: Christian Melki @ 2024-02-25 14:49 UTC (permalink / raw)
To: ptxdist
So, after asking questions about the default multithread mode
of xz 5.6.0 and determinism, it turned out that there
is a common misconception about determinism in xz.
XZ compression varies between the two compressors. Ie, the single
thread and multi thread variant. T0 always uses the multithread
one, even if used on one core. But T1 uses the singlethread
compressor.
This has caused a very common misconception about determinism
variability with number of threads. I've seen this discussion
on a lot of places. So albeit true, it only applies to the
distinct compressors, not how many threads are used in
multithread mode.
Now after giving this some thought I've decided to reenable
threaded XZ, using T0. T0 should produce the same results always,
on any target. Even unicore machines. And with the multithread
compressor now being the default, users should end up with the
same results unless they are actively using T1.
Signed-off-by: Christian Melki <christian.melki@t2data.com>
---
platforms/image-root-cpio.in | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/platforms/image-root-cpio.in b/platforms/image-root-cpio.in
index 0338175a0..546dda7fb 100644
--- a/platforms/image-root-cpio.in
+++ b/platforms/image-root-cpio.in
@@ -84,7 +84,7 @@ config IMAGE_ROOT_CPIO_COMPRESSION_UTIL
default "" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_NONE
default "gzip" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_GZ
default "zstd -T0 -19" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_ZSTD
- default "xz --check=crc32" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
+ default "xz -T0 --check=crc32" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
default "lzop" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_LZOP
endif
--
2.34.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [ptxdist] [PATCH] image-root-cpio.in: Use multithreaded XZ (revisited).
2024-02-25 14:49 [ptxdist] [PATCH] image-root-cpio.in: Use multithreaded XZ (revisited) Christian Melki
@ 2024-03-07 16:04 ` Michael Olbrich
2024-03-08 18:04 ` Christian Melki
0 siblings, 1 reply; 3+ messages in thread
From: Michael Olbrich @ 2024-03-07 16:04 UTC (permalink / raw)
To: Christian Melki; +Cc: ptxdist
Hi,
On Sun, Feb 25, 2024 at 03:49:06PM +0100, Christian Melki wrote:
> So, after asking questions about the default multithread mode
> of xz 5.6.0 and determinism, it turned out that there
> is a common misconception about determinism in xz.
>
> XZ compression varies between the two compressors. Ie, the single
> thread and multi thread variant. T0 always uses the multithread
> one, even if used on one core. But T1 uses the singlethread
> compressor.
>
> This has caused a very common misconception about determinism
> variability with number of threads. I've seen this discussion
> on a lot of places. So albeit true, it only applies to the
> distinct compressors, not how many threads are used in
> multithread mode.
>
> Now after giving this some thought I've decided to reenable
> threaded XZ, using T0. T0 should produce the same results always,
> on any target. Even unicore machines. And with the multithread
> compressor now being the default, users should end up with the
> same results unless they are actively using T1.
So if I understand it correctly then multithreaded compression results i a
slightly larger output file. But it also means that it can be extracted
with multiple threads.
The question is: will that actually happen? The initramfs is extracted by
the kernel and I'm not sure if it supports that.
Michael
> Signed-off-by: Christian Melki <christian.melki@t2data.com>
> ---
> platforms/image-root-cpio.in | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/platforms/image-root-cpio.in b/platforms/image-root-cpio.in
> index 0338175a0..546dda7fb 100644
> --- a/platforms/image-root-cpio.in
> +++ b/platforms/image-root-cpio.in
> @@ -84,7 +84,7 @@ config IMAGE_ROOT_CPIO_COMPRESSION_UTIL
> default "" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_NONE
> default "gzip" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_GZ
> default "zstd -T0 -19" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_ZSTD
> - default "xz --check=crc32" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
> + default "xz -T0 --check=crc32" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
> default "lzop" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_LZOP
>
> endif
> --
> 2.34.1
>
>
>
--
Pengutronix e.K. | |
Steuerwalder Str. 21 | http://www.pengutronix.de/ |
31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [ptxdist] [PATCH] image-root-cpio.in: Use multithreaded XZ (revisited).
2024-03-07 16:04 ` Michael Olbrich
@ 2024-03-08 18:04 ` Christian Melki
0 siblings, 0 replies; 3+ messages in thread
From: Christian Melki @ 2024-03-08 18:04 UTC (permalink / raw)
To: Michael Olbrich; +Cc: ptxdist
On 3/7/24 17:04, Michael Olbrich wrote:
> Hi,
>
> On Sun, Feb 25, 2024 at 03:49:06PM +0100, Christian Melki wrote:
>> So, after asking questions about the default multithread mode
>> of xz 5.6.0 and determinism, it turned out that there
>> is a common misconception about determinism in xz.
>>
>> XZ compression varies between the two compressors. Ie, the single
>> thread and multi thread variant. T0 always uses the multithread
>> one, even if used on one core. But T1 uses the singlethread
>> compressor.
>>
>> This has caused a very common misconception about determinism
>> variability with number of threads. I've seen this discussion
>> on a lot of places. So albeit true, it only applies to the
>> distinct compressors, not how many threads are used in
>> multithread mode.
>>
>> Now after giving this some thought I've decided to reenable
>> threaded XZ, using T0. T0 should produce the same results always,
>> on any target. Even unicore machines. And with the multithread
>> compressor now being the default, users should end up with the
>> same results unless they are actively using T1.
>
> So if I understand it correctly then multithreaded compression results i a
> slightly larger output file. But it also means that it can be extracted
> with multiple threads.
> The question is: will that actually happen? The initramfs is extracted by
> the kernel and I'm not sure if it supports that.
>
> Michael
>
Not sure I follow. You're asking if a multithreaded compressed archive
can be decompressed by the in kernel (single thread, assumed) decompressor?
It would seem it's very bad if it wasn't capable of doing so, so I never
think about that. I just assume it.
Regardless. You may drop this if you like, regarding slightly larger
archives. It's a very trivial local change for those who thinks it's useful.
Regards,
Christian
>> Signed-off-by: Christian Melki <christian.melki@t2data.com>
>> ---
>> platforms/image-root-cpio.in | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/platforms/image-root-cpio.in b/platforms/image-root-cpio.in
>> index 0338175a0..546dda7fb 100644
>> --- a/platforms/image-root-cpio.in
>> +++ b/platforms/image-root-cpio.in
>> @@ -84,7 +84,7 @@ config IMAGE_ROOT_CPIO_COMPRESSION_UTIL
>> default "" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_NONE
>> default "gzip" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_GZ
>> default "zstd -T0 -19" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_ZSTD
>> - default "xz --check=crc32" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
>> + default "xz -T0 --check=crc32" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
>> default "lzop" if IMAGE_ROOT_CPIO_COMPRESSION_MODE_LZOP
>>
>> endif
>> --
>> 2.34.1
>>
>>
>>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2024-03-08 18:07 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-02-25 14:49 [ptxdist] [PATCH] image-root-cpio.in: Use multithreaded XZ (revisited) Christian Melki
2024-03-07 16:04 ` Michael Olbrich
2024-03-08 18:04 ` Christian Melki
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox