archives

Hello! I would like to propose an optimization for XZ decompression. Currently, XZ decompression is performed sequentially in a single thread. We have implemented a concurrent parallel block decompressor in a fork of the underlying dependency at (which functions as a drop-in replacement). By leveraging the XZ format's native index block boundaries, the parallel decompressor parses the index backwards in O(1) time and decompresses independent blocks concurrently using a worker pool. To keep memory utilization bounded and prevent Garbage Collector overhead, we also introduced pooling for both decompressed block buffers and the large LZMA decoder dictionary slices. Benchmarks (on a 2-core / 4-thread CPU with a 20MB payload): Vanilla decompressor: ~14 MB/s Optimized sequential decompressor: ~30 MB/s Optimized parallel decompressor:** ~70 MB/s On systems with 4, 8, or more physical cores, the throughput scales near-linearly, easily exceeding 120+ MB/s. We can utilize this optimization when the input source implements and the total compressed stream size is known. Integration Instruction To utilize the parallel reader within your Go code when dealing with random-access inputs, you can check if the underlying stream supports seeking and pass it to the parallel decompressor.

Summary This pull request implements multi-threaded, concurrent XZ decompression for independent blocks when the input stream supports random access. This change is a direct response to the issue described in #75. Performance Impact By leveraging block-level parallelism, decompression performance scales near-linearly with available CPU cores on multi-block XZ files. Sequential baseline: ~14–32 MB/s Parallel decompression (multi-core): Up to 70–80 MB/s (on modern quad-core CPUs) Changes & Implementation Details 1. Dependency Switch: Replaced the original dependency with the fork. The fork includes a performance-optimized sequential decoder (using register caching and manual inlining) and the new worker pool implementation. 2. Interface Check: The implementation in now checks if the incoming implements (supporting both and ). 3. Parallel Activation: When seek/read-at capabilities are verified, we calculate the stream size, account for potential starting offsets (by wrapping with ), and launch the concurrent . 4. Resilient Fallback: If the stream is unseekable (standard sequential pipes) or initialization fails, the decoder safely falls back to standard sequential decompression without errors. 5. Testing: Added a comprehensive suite of unit tests in covering happy paths, offset streams, sequential fallbacks, seek error recovery, and partial reads. Fixes #75

What would you like to have changed? I think this mostly applied to , but may apply to others as well. There's currently a way to skip preserving UID and GID during the archive creation process. I'd like to request the same for extraction as well. Why is this feature a useful, necessary, and/or important addition to this project? I'm using it inside a project where I don't really care about the UID and GID because they're modified later anyway. My terminal is flooded with a lot of warnings which I'd rather not see. What alternatives are there, or what are you doing in the meantime to work around the lack of this feature? I'm thinking about processing the output, and potentially filtering these warnings out. Please link to any relevant issues, pull requests, or other discussions. N/A