Re: [ptxdist] [PATCH] image-root-cpio.in: Use multithreaded XZ (revisited).

[Christian Melki <christian.melki@t2data.com>]

On 3/7/24 17:04, Michael Olbrich wrote:
> Hi,
> 
> On Sun, Feb 25, 2024 at 03:49:06PM +0100, Christian Melki wrote:
>> So, after asking questions about the default multithread mode
>> of xz 5.6.0 and determinism, it turned out that there
>> is a common misconception about determinism in xz.
>>
>> XZ compression varies between the two compressors. Ie, the single
>> thread and multi thread variant. T0 always uses the multithread
>> one, even if used on one core. But T1 uses the singlethread
>> compressor.
>>
>> This has caused a very common misconception about determinism
>> variability with number of threads. I've seen this discussion
>> on a lot of places. So albeit true, it only applies to the
>> distinct compressors, not how many threads are used in
>> multithread mode.
>>
>> Now after giving this some thought I've decided to reenable
>> threaded XZ, using T0. T0 should produce the same results always,
>> on any target. Even unicore machines. And with the multithread
>> compressor now being the default, users should end up with the
>> same results unless they are actively using T1.
> 
> So if I understand it correctly then multithreaded compression results i a
> slightly larger output file. But it also means that it can be extracted
> with multiple threads.
> The question is: will that actually happen? The initramfs is extracted by
> the kernel and I'm not sure if it supports that.
> 
> Michael
> 

Not sure I follow. You're asking if a multithreaded compressed archive
can be decompressed by the in kernel (single thread, assumed) decompressor?
It would seem it's very bad if it wasn't capable of doing so, so I never
think about that. I just assume it.

Regardless. You may drop this if you like, regarding slightly larger
archives. It's a very trivial local change for those who thinks it's useful.

Regards,
Christian

>> Signed-off-by: Christian Melki <christian.melki@t2data.com>
>> ---
>>  platforms/image-root-cpio.in | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/platforms/image-root-cpio.in b/platforms/image-root-cpio.in
>> index 0338175a0..546dda7fb 100644
>> --- a/platforms/image-root-cpio.in
>> +++ b/platforms/image-root-cpio.in
>> @@ -84,7 +84,7 @@ config IMAGE_ROOT_CPIO_COMPRESSION_UTIL
>>  	default ""			if IMAGE_ROOT_CPIO_COMPRESSION_MODE_NONE
>>  	default "gzip"			if IMAGE_ROOT_CPIO_COMPRESSION_MODE_GZ
>>  	default "zstd -T0 -19"		if IMAGE_ROOT_CPIO_COMPRESSION_MODE_ZSTD
>> -	default "xz --check=crc32"	if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
>> +	default "xz -T0 --check=crc32"	if IMAGE_ROOT_CPIO_COMPRESSION_MODE_XZ
>>  	default "lzop"			if IMAGE_ROOT_CPIO_COMPRESSION_MODE_LZOP
>>  
>>  endif
>> -- 
>> 2.34.1
>>
>>
>>
>