1.. SPDX-License-Identifier: 0BSD 2 3============================ 4XZ data compression in Linux 5============================ 6 7Introduction 8============ 9 10XZ is a general purpose data compression format with high compression 11ratio. The XZ decompressor in Linux is called XZ Embedded. It supports 12the LZMA2 filter and optionally also Branch/Call/Jump (BCJ) filters 13for executable code. CRC32 is supported for integrity checking. The 14home page of XZ Embedded is at <https://tukaani.org/xz/embedded.html>. 15There you can find the latest version which includes a few optional 16extra features that aren't required in the Linux kernel and information 17about using the code outside the Linux kernel. 18 19For userspace, XZ Utils provide a zlib-like compression library 20and a gzip-like command line tool. XZ Utils can be downloaded from 21<https://tukaani.org/xz/>. 22 23XZ related components in the kernel 24=================================== 25 26The xz_dec module provides XZ decompressor with single-call (buffer 27to buffer) and multi-call (stateful) APIs. The usage of the xz_dec 28module is documented in include/linux/xz.h. 29 30For decompressing the kernel image, initramfs, and initrd, there 31is a wrapper function in lib/decompress_unxz.c. Its API is the 32same as in other decompress_*.c files, which is defined in 33include/linux/decompress/generic.h. 34 35For kernel makefiles, three commands are provided for use with 36$(call if_changed). They require the xz tool from XZ Utils. 37 38 - $(call if_changed,xzkern) is for compressing the kernel image. 39 It runs the script scripts/xz_wrap.sh which uses arch-optimized 40 options and a big LZMA2 dictionary. 41 42 - $(call if_changed,xzkern_with_size) is like xzkern above but this 43 also appends a four-byte trailer containing the uncompressed size 44 of the file, which is needed by the boot code on some archs. 45 46 - Other things can be compressed with $(call if_needed,xzmisc) 47 which will use no BCJ filter and 1 MiB LZMA2 dictionary. 48 49Notes on compression options 50============================ 51 52Since the XZ Embedded supports only streams with no integrity check or 53CRC32, make sure that you don't use some other integrity check type 54when encoding files that are supposed to be decoded by the kernel. With 55liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32 56when encoding. With the xz command line tool, use --check=none or 57--check=crc32 to override the default --check=crc64. 58 59Using CRC32 is strongly recommended unless there is some other layer 60which will verify the integrity of the uncompressed data anyway. 61Double checking the integrity would probably be waste of CPU cycles. 62Note that the headers will always have a CRC32 which will be validated 63by the decoder; you can only change the integrity check type (or 64disable it) for the actual uncompressed data. 65 66In userspace, LZMA2 is typically used with dictionary sizes of several 67megabytes. The decoder needs to have the dictionary in RAM: 68 69 - In multi-call mode the dictionary is allocated as part of the 70 decoder state. The reasonable maximum dictionary size for in-kernel 71 use will depend on the target hardware: a few megabytes should be 72 fine for desktop systems while 64 KiB to 1 MiB might be more 73 appropriate on some embedded systems. 74 75 - In single-call mode the output buffer is used as the dictionary 76 buffer. That is, the size of the dictionary doesn't affect the 77 decompressor memory usage at all. Only the base data structures 78 are allocated which take a little less than 30 KiB of memory. 79 For the best compression, the dictionary should be at least 80 as big as the uncompressed data. A notable example of single-call 81 mode is decompressing the kernel itself (except on PowerPC). 82 83The compression presets in XZ Utils may not be optimal when creating 84files for the kernel, so don't hesitate to use custom settings to, 85for example, set the dictionary size. Also, xz may produce a smaller 86file in single-threaded mode so setting that explicitly is recommended. 87Example:: 88 89 xz --threads=1 --check=crc32 --lzma2=dict=512KiB inputfile 90