1Zstandard library files 2================================ 3 4The __lib__ directory is split into several sub-directories, 5in order to make it easier to select or exclude features. 6 7 8#### Building 9 10`Makefile` script is provided, supporting [Makefile conventions](https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html#Makefile-Conventions), 11including commands variables, staged install, directory variables and standard targets. 12- `make` : generates both static and dynamic libraries 13- `make install` : install libraries and headers in target system directories 14 15`libzstd` default scope is pretty large, including compression, decompression, dictionary builder, 16and support for decoding legacy formats >= v0.5.0. 17The scope can be reduced on demand (see paragraph _modular build_). 18 19 20#### Multithreading support 21 22When building with `make`, by default the dynamic library is multithreaded and static library is single-threaded (for compatibility reasons). 23 24Enabling multithreading requires 2 conditions : 25- set build macro `ZSTD_MULTITHREAD` (`-DZSTD_MULTITHREAD` for `gcc`) 26- for POSIX systems : compile with pthread (`-pthread` compilation flag for `gcc`) 27 28For convenience, we provide a build target to generate multi and single threaded libraries: 29- Force enable multithreading on both dynamic and static libraries by appending `-mt` to the target, e.g. `make lib-mt`. 30- Force disable multithreading on both dynamic and static libraries by appending `-nomt` to the target, e.g. `make lib-nomt`. 31- By default, as mentioned before, dynamic library is multithreaded, and static library is single-threaded, e.g. `make lib`. 32 33When linking a POSIX program with a multithreaded version of `libzstd`, 34note that it's necessary to invoke the `-pthread` flag during link stage. 35 36Multithreading capabilities are exposed 37via the [advanced API defined in `lib/zstd.h`](https://github.com/facebook/zstd/blob/v1.4.3/lib/zstd.h#L351). 38 39 40#### API 41 42Zstandard's stable API is exposed within [lib/zstd.h](zstd.h). 43 44 45#### Advanced API 46 47Optional advanced features are exposed via : 48 49- `lib/zstd_errors.h` : translates `size_t` function results 50 into a `ZSTD_ErrorCode`, for accurate error handling. 51 52- `ZSTD_STATIC_LINKING_ONLY` : if this macro is defined _before_ including `zstd.h`, 53 it unlocks access to the experimental API, 54 exposed in the second part of `zstd.h`. 55 All definitions in the experimental APIs are unstable, 56 they may still change in the future, or even be removed. 57 As a consequence, experimental definitions shall ___never be used with dynamic library___ ! 58 Only static linking is allowed. 59 60 61#### Modular build 62 63It's possible to compile only a limited set of features within `libzstd`. 64The file structure is designed to make this selection manually achievable for any build system : 65 66- Directory `lib/common` is always required, for all variants. 67 68- Compression source code lies in `lib/compress` 69 70- Decompression source code lies in `lib/decompress` 71 72- It's possible to include only `compress` or only `decompress`, they don't depend on each other. 73 74- `lib/dictBuilder` : makes it possible to generate dictionaries from a set of samples. 75 The API is exposed in `lib/dictBuilder/zdict.h`. 76 This module depends on both `lib/common` and `lib/compress` . 77 78- `lib/legacy` : makes it possible to decompress legacy zstd formats, starting from `v0.1.0`. 79 This module depends on `lib/common` and `lib/decompress`. 80 To enable this feature, define `ZSTD_LEGACY_SUPPORT` during compilation. 81 Specifying a number limits versions supported to that version onward. 82 For example, `ZSTD_LEGACY_SUPPORT=2` means : "support legacy formats >= v0.2.0". 83 Conversely, `ZSTD_LEGACY_SUPPORT=0` means "do __not__ support legacy formats". 84 By default, this build macro is set as `ZSTD_LEGACY_SUPPORT=5`. 85 Decoding supported legacy format is a transparent capability triggered within decompression functions. 86 It's also allowed to invoke legacy API directly, exposed in `lib/legacy/zstd_legacy.h`. 87 Each version does also provide its own set of advanced API. 88 For example, advanced API for version `v0.4` is exposed in `lib/legacy/zstd_v04.h` . 89 90- While invoking `make libzstd`, it's possible to define build macros 91 `ZSTD_LIB_COMPRESSION`, `ZSTD_LIB_DECOMPRESSION`, `ZSTD_LIB_DICTBUILDER`, 92 and `ZSTD_LIB_DEPRECATED` as `0` to forgo compilation of the 93 corresponding features. This will also disable compilation of all 94 dependencies (e.g. `ZSTD_LIB_COMPRESSION=0` will also disable 95 dictBuilder). 96 97- There are a number of options that can help minimize the binary size of 98 `libzstd`. 99 100 The first step is to select the components needed (using the above-described 101 `ZSTD_LIB_COMPRESSION` etc.). 102 103 The next step is to set `ZSTD_LIB_MINIFY` to `1` when invoking `make`. This 104 disables various optional components and changes the compilation flags to 105 prioritize space-saving. 106 107 Detailed options: Zstandard's code and build environment is set up by default 108 to optimize above all else for performance. In pursuit of this goal, Zstandard 109 makes significant trade-offs in code size. For example, Zstandard often has 110 more than one implementation of a particular component, with each 111 implementation optimized for different scenarios. For example, the Huffman 112 decoder has complementary implementations that decode the stream one symbol at 113 a time or two symbols at a time. Zstd normally includes both (and dispatches 114 between them at runtime), but by defining `HUF_FORCE_DECOMPRESS_X1` or 115 `HUF_FORCE_DECOMPRESS_X2`, you can force the use of one or the other, avoiding 116 compilation of the other. Similarly, `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT` 117 and `ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG` force the compilation and use of 118 only one or the other of two decompression implementations. The smallest 119 binary is achieved by using `HUF_FORCE_DECOMPRESS_X1` and 120 `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT` (implied by `ZSTD_LIB_MINIFY`). 121 122 On the compressor side, Zstd's compression levels map to several internal 123 strategies. In environments where the higher compression levels aren't used, 124 it is possible to exclude all but the fastest strategy with 125 `ZSTD_LIB_EXCLUDE_COMPRESSORS_DFAST_AND_UP=1`. (Note that this will change 126 the behavior of the default compression level.) Or if you want to retain the 127 default compressor as well, you can set 128 `ZSTD_LIB_EXCLUDE_COMPRESSORS_GREEDY_AND_UP=1`, at the cost of an additional 129 ~20KB or so. 130 131 For squeezing the last ounce of size out, you can also define 132 `ZSTD_NO_INLINE`, which disables inlining, and `ZSTD_STRIP_ERROR_STRINGS`, 133 which removes the error messages that are otherwise returned by 134 `ZSTD_getErrorName` (implied by `ZSTD_LIB_MINIFY`). 135 136 Finally, when integrating into your application, make sure you're doing link- 137 time optimization and unused symbol garbage collection (via some combination of, 138 e.g., `-flto`, `-ffat-lto-objects`, `-fuse-linker-plugin`, 139 `-ffunction-sections`, `-fdata-sections`, `-fmerge-all-constants`, 140 `-Wl,--gc-sections`, `-Wl,-z,norelro`, and an archiver that understands 141 the compiler's intermediate representation, e.g., `AR=gcc-ar`). Consult your 142 compiler's documentation. 143 144- While invoking `make libzstd`, the build macro `ZSTD_LEGACY_MULTITHREADED_API=1` 145 will expose the deprecated `ZSTDMT` API exposed by `zstdmt_compress.h` in 146 the shared library, which is now hidden by default. 147 148- The build macro `DYNAMIC_BMI2` can be set to 1 or 0 in order to generate binaries 149 which can detect at runtime the presence of BMI2 instructions, and use them only if present. 150 These instructions contribute to better performance, notably on the decoder side. 151 By default, this feature is automatically enabled on detecting 152 the right instruction set (x64) and compiler (clang or gcc >= 5). 153 It's obviously disabled for different cpus, 154 or when BMI2 instruction set is _required_ by the compiler command line 155 (in this case, only the BMI2 code path is generated). 156 Setting this macro will either force to generate the BMI2 dispatcher (1) 157 or prevent it (0). It overrides automatic detection. 158 159- The build macro `ZSTD_NO_UNUSED_FUNCTIONS` can be defined to hide the definitions of functions 160 that zstd does not use. Not all unused functions are hidden, but they can be if needed. 161 Currently, this macro will hide function definitions in FSE and HUF that use an excessive 162 amount of stack space. 163 164- The build macro `ZSTD_NO_INTRINSICS` can be defined to disable all explicit intrinsics. 165 Compiler builtins are still used. 166 167- The build macro `ZSTD_DECODER_INTERNAL_BUFFER` can be set to control 168 the amount of extra memory used during decompression to store literals. 169 This defaults to 64kB. Reducing this value reduces the memory footprint of 170 `ZSTD_DCtx` decompression contexts, 171 but might also result in a small decompression speed cost. 172 173- The C compiler macros `ZSTDLIB_VISIBLE`, `ZSTDERRORLIB_VISIBLE` and `ZDICTLIB_VISIBLE` 174 can be overridden to control the visibility of zstd's API. Additionally, 175 `ZSTDLIB_STATIC_API` and `ZDICTLIB_STATIC_API` can be overridden to control the visibility 176 of zstd's static API. Specifically, it can be set to `ZSTDLIB_HIDDEN` to hide the symbols 177 from the shared library. These macros default to `ZSTDLIB_VISIBILITY`, 178 `ZSTDERRORLIB_VSIBILITY`, and `ZDICTLIB_VISIBILITY` if unset, for backwards compatibility 179 with the old macro names. 180 181- The C compiler macro `HUF_DISABLE_FAST_DECODE` disables the newer Huffman fast C 182 and assembly decoding loops. You may want to use this macro if these loops are 183 slower on your platform. 184 185#### Windows : using MinGW+MSYS to create DLL 186 187DLL can be created using MinGW+MSYS with the `make libzstd` command. 188This command creates `dll\libzstd.dll` and the import library `dll\libzstd.lib`. 189The import library is only required with Visual C++. 190The header file `zstd.h` and the dynamic library `dll\libzstd.dll` are required to 191compile a project using gcc/MinGW. 192The dynamic library has to be added to linking options. 193It means that if a project that uses ZSTD consists of a single `test-dll.c` 194file it should be linked with `dll\libzstd.dll`. For example: 195``` 196 gcc $(CFLAGS) -Iinclude/ test-dll.c -o test-dll dll\libzstd.dll 197``` 198The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`. 199 200 201#### Advanced Build options 202 203The build system requires a hash function in order to 204separate object files created with different compilation flags. 205By default, it tries to use `md5sum` or equivalent. 206The hash function can be manually switched by setting the `HASH` variable. 207For example : `make HASH=xxhsum` 208The hash function needs to generate at least 64-bit using hexadecimal format. 209When no hash function is found, 210the Makefile just generates all object files into the same default directory, 211irrespective of compilation flags. 212This functionality only matters if `libzstd` is compiled multiple times 213with different build flags. 214 215The build directory, where object files are stored 216can also be manually controlled using variable `BUILD_DIR`, 217for example `make BUILD_DIR=objectDir/v1`. 218In which case, the hash function doesn't matter. 219 220 221#### Deprecated API 222 223Obsolete API on their way out are stored in directory `lib/deprecated`. 224At this stage, it contains older streaming prototypes, in `lib/deprecated/zbuff.h`. 225These prototypes will be removed in some future version. 226Consider migrating code towards supported streaming API exposed in `zstd.h`. 227 228 229#### Miscellaneous 230 231The other files are not source code. There are : 232 233 - `BUCK` : support for `buck` build system (https://buckbuild.com/) 234 - `Makefile` : `make` script to build and install zstd library (static and dynamic) 235 - `README.md` : this file 236 - `dll/` : resources directory for Windows compilation 237 - `libzstd.pc.in` : script for `pkg-config` (used in `make install`) 238