Zstandard 이전 버전 및 전체 버전

Zstandard v1.5.7 is a significant release, featuring over 500 commits accumulated over the past year. This update brings enhancements across various domains, including performance, stability, and functionality, and is particularly recommended for 32-bit users due to a bug fix detailed below.
edit: man pages were not properly updated in this package, and are still pointing to v1.5.6. If you want the v1.5.7 version, they are updated in dev branch, and can be cherry-picked using commit 6af3842. Alternatively, they can also be built locally using make manual and make -C programs man.
Performance Enhancements
Enhanced Compression Speed for Small Data Blocks
The compression speed for small data blocks has been notably improved at fast compression levels, thanks to contributions from @TocarIP, further extended in #4165. Below are benchmark results illustrating the performance improvements for level 1 compression on the Silesia corpus, segmented into different block sizes:
──────────┬────────┬────────┬───────
Block Size│v1.5.6 │v1.5.7 │Improv.
──────────┼────────┼────────┼───────
4 KB │280 MB/s│310 MB/s│+10%
──────────┼────────┼────────┼───────
8 KB │328 MB/s│364 MB/s│+11%
──────────┼────────┼────────┼───────
16 KB │360 MB/s│396 MB/s│+10%
──────────┼────────┼────────┼───────
32 KB │383 MB/s│458 MB/s│+20%
──────────┼────────┼────────┼───────
64 KB │423 MB/s│474 MB/s│+12%
──────────┼────────┼────────┼───────
128 KB │460 MB/s│485 MB/s│+5%
──────────┴────────┴────────┴───────
These improvements are particularly beneficial in data centers and databases, where data is often divided into small segments to enhance seekability. A notable example is RocksDB, which is frequently configured with block sizes around 16 KB. Such systems will experience an immediate performance boost by updating their compression library to v1.5.7.
Additionally, the speed improvements partially extend to dictionary compression, with observed gains ranging from none to +15%.
Substantial --patch-from improvements
The --patch-from functionality of the zstd CLI allows for differential compression of a file given an older version, similar to bsdiff. While compression levels impact processing speed and compression ratio as expected, at high compression levels (18+), speed was previously reduced to uncomfortable levels.
v1.5.7 largely mitigates the speed impact of high compression levels 18+, as illustrated in the following scenario, which compresses linux-6.13.tar as a difference from linux-6.12.tar using 6 threads:
─────┬──────┬──────┬───────
level│v1.5.6│v1.5.7│Improv.
─────┼──────┼──────┼───────
18 │210 s │42.7 s│x4.9
─────┼──────┼──────┼───────
19 │273 s │61.8 s│x4.4
─────┼──────┼──────┼───────
20 │310 s │82.3 s│x3.8
─────┼──────┼──────┼───────
21 │365 s │120 s │x3.1
─────┴──────┴──────┴───────
Additionally, compression ratio of --patch-from is also substantially improved across the board, including higher compression levels:
─────┬─────────┬─────────┬───────
level│v1.5.6 │v1.5.7 │Improv.
─────┼─────────┼─────────┼───────
15 │6,383,341│6,003,514│6.3%
─────┼─────────┼─────────┼───────
16 │5,909,932│5,421,444│9.0%
─────┼─────────┼─────────┼───────
17 │5,613,879│5,214,734│7.7%
─────┼─────────┼─────────┼───────
18 │5,584,219│5,075,507│10.0%
─────┼─────────┼─────────┼───────
19 │5,322,658│4,724,116│12.6%
─────┼─────────┼─────────┼───────
20 │5,097,236│4,583,435│11.2%
─────┼─────────┼─────────┼───────
21 │4,648,583│4,297,672│8.1%
─────┼─────────┼─────────┼───────
22 │4,560,051│4,225,283│7.9%
─────┼─────────┼─────────┼───────
--max│N/A │3,999,476│
─────┴─────────┴─────────┴───────
Compression ratio improvements for large files
The compression ratio has been enhanced slightly for large data across all compression levels, thanks to a refined approach in selecting block boundaries. This improvement is more pronounced at lower compression levels (≤ 15), where the previous boundary selection method was relatively basic. In contrast, higher compression levels (16+) already employed

Change Log
- fix: fix rare corruption bug affecting the high compression mode, reported by @danlark1 (#3517, @terrelln)
- perf: improve mid-level compression speed (#3529, #3533, #3543, @yoniko and #3552, @terrelln)
- lib: deprecated bufferless block-level API (#3534) by @terrelln
- cli: mmap large dictionaries to save memory, by @daniellerozenblit
- cli: improve speed of --patch-from mode (~+50%) (#3545) by @daniellerozenblit
- cli: improve i/o speed (~+10%) when processing lots of small files (#3479) by @felixhandte
- cli: zstd no longer crashes when requested to write into write-protected directory (#3541) by @felixhandte
- cli: fix decompression into block device using -o (#3584, @Cyan4973) reported by @georgmu
- build: fix zstd CLI compiled with lzma support but not zlib support (#3494) by @Hello71
- build: fix cmake does no longer require 3.18 as minimum version (#3510) by @kou
- build: fix MSVC+ClangCL linking issue (#3569) by @tru
- build: fix zstd-dll, version of zstd CLI that links to the dynamic library (#3496) by @yoniko
- build: fix MSVC warnings (#3495) by @embg
- doc: updated zstd specification to clarify corner cases, by @Cyan4973
- doc: document how to create fat binaries for macos (#3568) by @rickmark
- misc: improve seekable format ingestion speed (~+100%) for very small chunk sizes (#3544) by @Cyan4973
- misc: tests/fullbench can benchmark multiple files (#3516) by @dloidolt
Full change list (auto-generated)
- Fix all MSVC warnings by @embg in #3495
- Fix zstd-dll build missing dependencies by @yoniko in #3496
- Bump github/codeql-action from 2.2.1 to 2.2.4 by @dependabot in #3503
- Github Action to generate Win64 artifacts by @Cyan4973 in #3491
- Use correct types in LZMA comp/decomp by @Hello71 in #3497
- Make Github workflows permissions read-only by default by @yoniko in #3488
- CI Workflow for external compressors dependencies by @yoniko in #3505
- Fix cli-tests issues by @daniellerozenblit in #3509
- Fix Permissions on Publish Release Artifacts Job by @felixhandte in #3511
- Use f-variants of chmod() and chown() by @felixhandte in #3479
- Don't require CMake 3.18 or later by @kou in #3510
- meson: always build the zstd binary when tests are enabled by @eli-schwartz in #3490
- [bug-fix] Fix rare corruption bug affecting the block splitter by @terrelln in #3517
- Clarify zstd specification for Huffman blocks by @Cyan4973 in #3514
- Fix typos found by codespell by @DimitriPapadopoulos in #3513
- Bump github/codeql-action from 2.2.4 to 2.2.5 by @dependabot in #3518
- fullbench with two files by @dloidolt in #3516
- Add initialization of clevel to static cdict (#3525) by @yoniko in #3527
- [linux-kernel] Fix assert definition by @terrelln in #3532
- Add ZSTD_set{C,F,}Params() helper functions by @terrelln in #3530
- Clarify dstCapacity requirements by @terrelln in #3531
- Mmap large dictionaries in patch-from mode by @daniellerozenblit in #3486
- added clarifications for sizes of compressed huffman blocks and streams. by @Cyan4973 in #3538
- Simplify benchmark unit invocation API from CLI by @Cyan4973 in #3526
- Avoid Segfault Caused by Calling setvbuf() on Null File Pointer by @felixhandte in #3541
- Pin Moar Action Dependencies by @felixhandte in #3542
- Improved seekable format ingestion speed for small frame size by @Cyan4973 in #3544
- Reduce RowHash's tag space size by x2 by @yoniko in #3543
- [Bugfix] row hash tries to match position 0 by @yoniko in #3548
- Bump github/codeql-action from 2.2.5 to 2.2.6 by @dependabot in #3549
- Add init once memory (#3528) by @yoniko in #3529
- Introduce salt into row hash (#3528 part 2) by @yoniko in #3533
- added documentation for the seekable format by @Cyan4973 in #3547
- patch-from speed optimization by @daniellerozenblit in #3545
- Deprecated bufferless and block level APIs by @terrelln in #3534
- added documentation for LDM + dictionary compatibility by @Cyan4973 in #3553
- Fix a bug in the CLI tests newline processing, then simplify it further by @ppentchev