how to speed up compression?

[jml75]

You could use pigz (parallel implementation of gzip) http://www.zlib.net/pigz/pigz-2.1.6.tar.gz as I did with great success.

The performance increase on a multicore system of pigz vs gzip, wich is a single core only program, is linear.

On one of my test systems (dual-quad 3Ghz Xeon with 8G of ram and a 4 disk 7200 SATA raid 5 array), I achived the following result :

22MBps with 1 core;
44MBps with 2 cores;
88MBps with 4 cores;
176MBps with 8 cores.

You get the picture.

But you could as well buy a gzip compression card like the AHA363-PCIe from AHA : http://www.aha.com/show_prod.php?id=36, wich will be more efficient, more powerfull and alot cheaper than getting the 32 core system you would need to achieve the same 625MBps compress/decompress throughtput this card can push.

This card cost around $1290 US according the rep from AHA.

Hope this helps.

Jonathan