xz compression program, run with defaults, can be so slow that you might wonder why anybody uses it. But on a modern machine,
xz -0 --threads=0 will run with less aggressive compression goals and use all available threads on a multi-core CPU, exceeding gzip performance by a good measure. One nice bonus with the xz format is that individual archives can be concatenated.
cat file*.xz > all-files.xz without decompressing first, and then unarchive in one step – nifty! You can even null-pad the space between concatenated files if you need to block-align the archive for random access on slow media. You might consider setting your XZ_DEFAULTS environment variable to use these higher performance options.
Let’s look at compressing this Postscript file:
$ ls -lh certificate5896072-3.ps -rw-r--r-- 1 user user 30M Nov 20 2012 certificate5896072-3.ps $ time gzip -9 -k certificate5896072-3.ps real 0m1.828s user 0m1.697s sys 0m0.029s $ ls -lh certificate5896072-3.ps.gz -rw-r--r-- 1 user user 28M Nov 20 2012 certificate5896072-3.ps.gz $ time xz -0 --threads=0 certificate5896072-3.ps real 0m1.250s user 0m8.781s sys 0m0.077s $ ls -lh certificate5896072-3.ps.xz -rw-r--r-- 1 user user 27M Nov 20 2012 certificate5896072-3.ps.xz $ time gzip -1 -k certificate5896072-3.ps real 0m1.365s user 0m1.330s sys 0m0.031s $ ls -lh certificate5896072-3.ps.gz -rw-r--r-- 1 user user 28M Nov 20 2012 certificate5896072-3.ps.gz $ time xz certificate5896072-3.ps real 0m1.974s user 0m9.843s sys 0m0.122s $ ls -lh certificate5896072-3.ps.xz -rw-r--r-- 1 user user 26M Nov 20 2012 certificate5896072-3.ps.xz $ time xz -9 --threads=0 certificate5896072-3.ps real 0m13.182s user 0m12.980s sys 0m0.190s $ ls -lh certificate5896072-3.ps.xz -rw-r--r-- 1 user user 23M Nov 20 2012 certificate5896072-3.ps.xz $ time xz -6 --threads=1 certificate5896072-3.ps real 0m13.615s user 0m13.494s sys 0m0.081s $ ls -lh certificate5896072-3.ps.xz -rw-r--r-- 1 user user 24M Nov 20 2012 certificate5896072-3.ps.xz
As you can see, the fastest xz exceeds the compression ratio of the slowest gzip and the speed of the fastest gzip, by using multiple cores (test machine is an 8-thread/4-core portable i7). Allowing slightly more time than gzip gives slighly better compression, and allowing significantly more time than gzip yields marginally better compression still. But note the last test – with the xz defaults, the time is the worst and the compression isn’t the best. In that case, compression is more than 10% better but the time is 10 times worse. These defaults will only hinder the adoption of xz, which is a shame because faster defaults would get more users and already provide better results than gzip on typical workloads on modern CPU’s.
UPDATE 20170825: Antonio Diaz Diaz asked me to look at lzip as well. Here are the results I got:
$ time lzip -1 --keep certificate5896072-3.ps real 0m10.923s user 0m9.993s sys 0m0.037s $ ls -lh certificate5896072-3.ps.lz -rw-rw-r-- 1 user user 26M Apr 6 2013 certificate5896072-3.ps.lz $ time lzip -9 --keep certificate5896072-3.ps real 0m16.348s user 0m16.126s sys 0m0.167s $ ls -lh certificate5896072-3.ps.lz -rw-rw-r-- 1 user user 23M Apr 6 2013 certificate5896072-3.ps.lz $ time xz -9 --threads=1 --keep certificate5896072-3.ps real 0m14.867s user 0m14.559s sys 0m0.220s $ ls -lh certificate5896072-3.ps.xz -rw-rw-r-- 1 user user 23M Apr 6 2013 certificate5896072-3.ps.xz
As you can see, compression is better than gzip and about the same as xz. xz beats lzip’s performance, both in single and (more obviously) multithreadded, while gzip is an order of magnitude faster than lzip, but offers less compression. Antonio’s lzip page points out that defects in busybox’s unxz prevent the catching of CRC errors, so you might want to keep that in mind if you’re running embedded. I would encourage the busybox developers to care about data integrity over performance..