Anteru's blog
  • Consulting
  • Research
    • Assisted environment probe placement
    • Assisted texture assignment
    • Error Metrics for Smart Image Refinement
    • High-Quality Shadows for Streaming Terrain Rendering
    • Hybrid Sample-based Surface Rendering
    • Interactive rendering of Giga-Particle Fluid Simulations
    • Quantitative Analysis of Voxel Raytracing Acceleration Structures
    • Real-time Hybrid Hair Rendering
    • Scalable rendering for very large meshes
    • Subpixel Reconstruction Antialiasing
    • Tiled light trees
  • About
  • Archive

Advent 2021: Zstandard

December 19, 2021
  • Programming
approximately 2 minutes to read

Yet another library that has appeared recently and really changed how I do things is Zstandard – a compression library. Compression is an interesting area of computer science which hasn’t seen much innovation for a long time. When I started programming, there were dozens of competing algorithms and tools: RAR, UHARC, ACE, ARJ, ZIP, and a few more exotic ones. After a while, things seemed to settle down on ZIP and 7z, with ZIP being the clear leader (ZIP using deflate compression – ZIP itself is just the container, but I’s pretty much universally used with deflate. The canonical library for that is zlib). More recently, a few new algorithms started to crop up: Brotli, LZ4, and Snappy. Those were attacking the compression problem from new angles, and I ended up using LZ4 a lot of compression of data sets as it is incredibly easy to add, but they all remained fairly niche.

Fast forward a few years and a new contender entered the ring: Zstd. Zstd provided a great blend of compression quality and performance, being as good as standard ZIP at much higher speed, or being dramatically faster at slightly lower compression ratios. Especially in the age of NVMe drives which can read at GiB/s rates, you really don’t want a compression algorithm which can’t decompress at that speed. Additionally, it’s scalable: It can go extremely fast at reduced compression, or achieve very high compression rates at significantly higher cost.

For me, Zstd replaced most uses of LZ4 and zlib by now. I compress data sets using Zstd as it’s much faster and has no downsides, by being a better algorithm. I also use it for backups and other long-term archival storage as it doesn’t seem to go away anytime soon. The next frontier here will be hopefully a replacement for ZIP (the package format!) as that one hasn’t aged very well, but it still ubiquitous and often times the only supported built-in format. I’m optimistic though – there’s a good reason to develop a new on-disk format now (if only to support the latest advances in compression) and I wouldn’t be surprised if we’ll see a ZIP replacement in the coming years. Till then, I’ll continue to use Zstd where I already can!

Previous post
Next post

Recent posts

  • Advent 2021: Open source
    Posted on 2021-12-24
  • Advent 2021: Blender
    Posted on 2021-12-23
  • Advent 2021: Visual Studio Code
    Posted on 2021-12-22
  • Advent 2021: Thunderbird
    Posted on 2021-12-21
  • Advent 2021: ZFS
    Posted on 2021-12-20
  • Older posts

Find me on the web

  • GitHub
  • GPU database
  • Projects

Follow me

NIV_Anteru

Contents © 2005-2022 | Anteru | Imprint/Impressum | Privacy policy/Datenschutz | Made with Liara