[ptxdist] [PATCH] image-root-cpio.in: Use multithreaded XZ (revisited).

[ptxdist] [PATCH] image-root-cpio.in: Use multithreaded XZ (revisited).

[Christian Melki]

So, after asking questions about the default multithread mode
of xz 5.6.0 and determinism, it turned out that there
is a common misconception about determinism in xz.

XZ compression varies between the two compressors. Ie, the single
thread and multi thread variant. T0 always uses the multithread
one, even if used on one core. But T1 uses the singlethread
compressor.

This has caused a very common misconception about determinism
variability with number of threads. I've seen this discussion
on a lot of places. So albeit true, it only applies to the
distinct compressors, not how many threads are used in
multithread mode.

Now after giving this some thought I've decided to reenable
threaded XZ, using T0. T0 should produce the same results always,
on any target. Even unicore machines. And with the multithread
compressor now being the default, users should end up with the
same results unless they are actively using T1.

Signed-off-by: Christian Melki <christian.melki@t2data.com>
---
 platforms/image-root-cpio.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/platforms/image-root-cpio.in b/platforms/image-root-cpio.in
index 0338175a0..546dda7fb 100644
--- a/platforms/image-root-cpio.in
+++ b/platforms/image-root-cpio.in
@@ -84,7 +84,7 @@ config IMAGE_ROOT_CPIO_COMPRESSION_UTIL
 	default ""			if IMAGE_ROOT_CPIO_COMPRESSION_MODE_NONE
 	default "gzip"			if IMAGE_ROOT_CPIO_COMPRESSION_MODE_GZ
 	default "zstd -T0 -19"		if IMAGE_ROOT_CPIO_COMPRESSION_MODE_ZSTD
-	default "xz --check=crc32"	if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
+	default "xz -T0 --check=crc32"	if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
 	default "lzop"			if IMAGE_ROOT_CPIO_COMPRESSION_MODE_LZOP
 
 endif
-- 
2.34.1

Re: [ptxdist] [PATCH] image-root-cpio.in: Use multithreaded XZ (revisited).

[Michael Olbrich]

Hi,

On Sun, Feb 25, 2024 at 03:49:06PM +0100, Christian Melki wrote:
> So, after asking questions about the default multithread mode
> of xz 5.6.0 and determinism, it turned out that there
> is a common misconception about determinism in xz.
> 
> XZ compression varies between the two compressors. Ie, the single
> thread and multi thread variant. T0 always uses the multithread
> one, even if used on one core. But T1 uses the singlethread
> compressor.
> 
> This has caused a very common misconception about determinism
> variability with number of threads. I've seen this discussion
> on a lot of places. So albeit true, it only applies to the
> distinct compressors, not how many threads are used in
> multithread mode.
> 
> Now after giving this some thought I've decided to reenable
> threaded XZ, using T0. T0 should produce the same results always,
> on any target. Even unicore machines. And with the multithread
> compressor now being the default, users should end up with the
> same results unless they are actively using T1.

So if I understand it correctly then multithreaded compression results i a
slightly larger output file. But it also means that it can be extracted
with multiple threads.
The question is: will that actually happen? The initramfs is extracted by
the kernel and I'm not sure if it supports that.

Michael

> Signed-off-by: Christian Melki <christian.melki@t2data.com>
> ---
>  platforms/image-root-cpio.in | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/platforms/image-root-cpio.in b/platforms/image-root-cpio.in
> index 0338175a0..546dda7fb 100644
> --- a/platforms/image-root-cpio.in
> +++ b/platforms/image-root-cpio.in
> @@ -84,7 +84,7 @@ config IMAGE_ROOT_CPIO_COMPRESSION_UTIL
>  	default ""			if IMAGE_ROOT_CPIO_COMPRESSION_MODE_NONE
>  	default "gzip"			if IMAGE_ROOT_CPIO_COMPRESSION_MODE_GZ
>  	default "zstd -T0 -19"		if IMAGE_ROOT_CPIO_COMPRESSION_MODE_ZSTD
> -	default "xz --check=crc32"	if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
> +	default "xz -T0 --check=crc32"	if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
>  	default "lzop"			if IMAGE_ROOT_CPIO_COMPRESSION_MODE_LZOP
>  
>  endif
> -- 
> 2.34.1
> 
> 
> 

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [ptxdist] [PATCH] image-root-cpio.in: Use multithreaded XZ (revisited).

[Christian Melki]

On 3/7/24 17:04, Michael Olbrich wrote:
> Hi,
> 
> On Sun, Feb 25, 2024 at 03:49:06PM +0100, Christian Melki wrote:
>> So, after asking questions about the default multithread mode
>> of xz 5.6.0 and determinism, it turned out that there
>> is a common misconception about determinism in xz.
>>
>> XZ compression varies between the two compressors. Ie, the single
>> thread and multi thread variant. T0 always uses the multithread
>> one, even if used on one core. But T1 uses the singlethread
>> compressor.
>>
>> This has caused a very common misconception about determinism
>> variability with number of threads. I've seen this discussion
>> on a lot of places. So albeit true, it only applies to the
>> distinct compressors, not how many threads are used in
>> multithread mode.
>>
>> Now after giving this some thought I've decided to reenable
>> threaded XZ, using T0. T0 should produce the same results always,
>> on any target. Even unicore machines. And with the multithread
>> compressor now being the default, users should end up with the
>> same results unless they are actively using T1.
> 
> So if I understand it correctly then multithreaded compression results i a
> slightly larger output file. But it also means that it can be extracted
> with multiple threads.
> The question is: will that actually happen? The initramfs is extracted by
> the kernel and I'm not sure if it supports that.
> 
> Michael
> 

Not sure I follow. You're asking if a multithreaded compressed archive
can be decompressed by the in kernel (single thread, assumed) decompressor?
It would seem it's very bad if it wasn't capable of doing so, so I never
think about that. I just assume it.

Regardless. You may drop this if you like, regarding slightly larger
archives. It's a very trivial local change for those who thinks it's useful.

Regards,
Christian

>> Signed-off-by: Christian Melki <christian.melki@t2data.com>
>> ---
>>  platforms/image-root-cpio.in | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/platforms/image-root-cpio.in b/platforms/image-root-cpio.in
>> index 0338175a0..546dda7fb 100644
>> --- a/platforms/image-root-cpio.in
>> +++ b/platforms/image-root-cpio.in
>> @@ -84,7 +84,7 @@ config IMAGE_ROOT_CPIO_COMPRESSION_UTIL
>>  	default ""			if IMAGE_ROOT_CPIO_COMPRESSION_MODE_NONE
>>  	default "gzip"			if IMAGE_ROOT_CPIO_COMPRESSION_MODE_GZ
>>  	default "zstd -T0 -19"		if IMAGE_ROOT_CPIO_COMPRESSION_MODE_ZSTD
>> -	default "xz --check=crc32"	if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
>> +	default "xz -T0 --check=crc32"	if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
>>  	default "lzop"			if IMAGE_ROOT_CPIO_COMPRESSION_MODE_LZOP
>>  
>>  endif
>> -- 
>> 2.34.1
>>
>>
>>
>