BZIP

Section: User Commands (1)
Updated: local
Index

NAME

bzip, bunzip - a block-sorting file compressor, v0.21

SYNOPSIS

bzip[ -cdfkvVL123456789 ][filenames ...]
bunzip[ -kvVL ][filenames ...]

DESCRIPTION

Bzipcompresses files using the Burrows-Wheeler-Fenwick block-sorting text compression algorithm. Compression is generally considerablybetter than that achieved by more conventional LZ77/LZ78-based compressors,and competitive with all but the best of the PPM family of statisticalcompressors.

The command-line options are deliberately very similar to those of GNU Gzip,but they are not identical.

Bzip expects a list of file names to follow the command-line flags. Each file is replaced by a compressed version of itself,with the name "original_name.bz".Each compressed file has the same modification date and permissionsas the corresponding original, so that these properties can be correctly restored at decompression time. File name handling isnaive in the sense that there is no mechanism for preservingoriginal file names, permissions and dates in filesystems which lack these concepts, or have serious file name lengthrestrictions, such as MS-DOS.

Bzipandbunzipwill not overwrite existing files; if you want this to happen,you should delete them first.

If no file names are specified,bzipcompresses from standard input to standard output.In this case,bzipwill decline to write compressed output to a terminal, asthis would be entirely incomprehensible and therefore pointless.

Bunzip(orbzip -d) decompresses and restores all specified files whose namesend in ".bz".Files without this suffix are ignored. Again, supplying no filenamescauses decompression from standard input to standard output.

You can also compress or decompress exactly one named file tothe standard output by giving the -c flag.

Compression is always performed, even if the compressed file isslightly larger than the original. The worst case expansion isfor files of zero length, which expand to seventeen bytes.Random data (including the output of most file compressors)is coded at about 8.1 bits per byte, giving an expansion of around 1%.

As a self-check for your protection,bzipuses 32-bit CRCs to make sure that the decompressedversion of a file is identical to the original. This guards against corruption of the compressed data,and against undetected bugs inbzip(hopefully very unlikely).The chances of data corruption going undetected is microscopic, about one chance in four billionfor each file processed. Be aware, though, that the checkoccurs upon decompression, so it can only tell you thatthat something is wrong. It can't help you recover theoriginal uncompressed data.

Return values: 1 for an abnormal exit, otherwise 0.

MEMORY MANAGEMENT

Bzipcompresses large files in blocks. The block size affects both the compression ratio achieved, and the amount of memory needed both forcompression and decompression. The flags -1 through -9specify the block size to be 100,000 bytes through 900,000 bytes(the default) respectively. At decompression-time, the block size used forcompression is read from the header of the compressed file, andbunzipthen allocates itself just enough memory to decompress the file.Since block sizes are stored in compressed files, it follows that the flags-1 to -9are irrelevant to and so ignored during decompression.Compression and decompression requirements, in bytes, can be estimated as:

Compression: 300k + ( 8 x block size )

Decompression: 6 x block size

The 300k constant is for a frequency-counttable, used in the sorting phase of compression.

Larger block sizes give rapidly diminishing marginal returns; mostof the compression comes from the first two or three hundred k of block size,a fact worth bearing in mind when using bzipon small machines. It is also important to appreciate that thedecompression memory requirement is set at compression-time by thechoice of block size. So, for example, if you are compressingfiles which you think might possibly be decompressed on a 4-megabytemachine, you might want to select a block size of 200k or 300k, sothe decompressor will draw 1200 kbytes or 1800 kbytes respectively,which is probably the limit of what's comfortable on a 4-meg machine.In general, though, you should try and use the largest block sizememory constraints allow. Compression and decompressionspeed is virtually unaffected by block size.

Another significant point applies to files which fit in a singleblock -- that means most files you'd encounter using a large block size. The amount of real memory touched is proportionalto the size of the file, since the file is smaller than a block.For example, compressing a file 20,000 bytes long with the flag-9will cause the compressor to allocate [by the formula, in practice alittle more] 7500k of memory, but only touch 300k + 20000 * 8 = 460kbytes of it. Similarly, the decompressor will allocate 5400k butonly touch 20000 * 6 = 120 kbytes.

Here is a table which summarises the maximum memory usage for different block sizes. Also recorded is the total compressedsize for 14 files of the Calgary Text Compression Corpustotalling 3,141,622 bytes. This column gives some feel for howcompression varies with block size. These figures tend to understatethe advantage of larger block sizes for larger files, since theCorpus is dominated by smaller files.

Compress Decompress Corpus
Flag usage usage Size

          -1      1100k       500k      905958
          -2      1900k      1000k      870646
          -3      2700k      1500k      853650
          -4      3500k      2000k      840140
          -5      4300k      2500k      838355
          -6      5100k      3000k      831695
          -7      5900k      3500k      827104
          -8      6700k      4000k      821652
          -9      7500k      4500k      821652

OPTIONS

-c: Compress or decompress to standard output. -c requires you to supplyexactly one file name, and this file is compressed or decompressedto standard out.
-d: Force decompression.Bzipandbunzipare really the same program, and the decision about whether tocompress or decompress is done on the basis of which name isused. This flag overrides that mechanism, and forcesbzipto decompress.
-f: The complement to -d: forces compression, regardless of the invokationname.
-k: Keep (don't delete) input files during compression or decompression.
-v: Verbose mode -- show the compression ratio for each file processed.
-V: Be very verbose. This spews out lots of information duringcompression which is primarily of interest for debugging purposes.
-L: Display the software license terms and conditions.
-1 to -9: Set the block size to 100 k, 200 k .. 900 k whencompressing. Has no effect when decompressing.See MEMORY MANAGEMENT above.

PERFORMANCE NOTES

The sorting phase of compression gathers together similar stringsin the file. Because of this, files containing very long runs of repeated symbols, like "aabaabaabaab ..." (repeatedseveral hundred times) may compress extraordinarily slowly.You can use the-V option to monitor progress in great detail, if you want.Decompression speed is unaffected. Such pathological casesseem rare in practice.

Incompressible or virtually-incompressible data may decompressrather more slowly than one would hope. This is due to naive implementation of the move-to-front coder, and of thefrequency tables for the arithmetic coder.

Decompression on Sun Sparc 1's (and other low-range Sparcs) can be slow, because of thelack of hardware implementations of integer multiply and dividein the SPARC v7 instruction set. The situation is much exacerbatedif bzipis compiled for a full SPARC v8 instruction set, since this causesthe machine to trap on each multiply and divide instruction.These traps take control to the relevant software emulationof the offending instruction, but it is much quicker for thecompiler simply to plant a call to the emulation routine.Moral: be careful how you compile bzipfor a Sparc. If you use GNU C, investigate the effects ofthe -msupersparc and -mcypress flags.

Wildcard expansion for Windows 95 and NT loses leading directoryinformation. For example, the pathspec "sources\*.c" is searched correctly for matching files, but the "sources\" bit is ignored whenthe files come to be processed, which means bzipwon't be able to find any of them. This is easy to fix; perhapssome enterprising soul will send me a patch?

CAVEATS

I/O error messages are not as helpful as they could be.Bziptries hard to detect I/O errors and exit cleanly, but thedetails of what the problem is sometimes seem rather misleading.

There is no -t option to test the integrity of a compressedfile. However, Unix folks can do the following:

bzip -dcV file.bz > /dev/null

which causes bzipto do a trial decompression of file.bz, throwing awaythe result. You'll be shown the computed and stored CRCs.If these are identical, the file is almost certainly OK --see the discussion above on CRCs for a definition of"almost certainly".If they're not, bzipwill complain loudly. Note that file.bz is left unchangedregardless of the outcome. Win95/NT folks can do the same, but/dev/null will have to be replaced with something suitable,perhaps NUL.

This manual page pertains to version 0.21 of bzip. It may well happen that some future version willuse a different compressed file format. If you try to decompress, using 0.21, a .bz file created with somefuture version which uses a different compressed file format,0.21 will complain that your file "is not a BZIP file".If that happens, you should obtain a more recent versionof bzipand use that to decompress the file.

AUTHOR

Julian Seward, sewardjAATTcs.man.ac.uk.

The ideas embodied in bzipare due to (at least) the following people:Michael Burrows and David Wheeler (for the block sorting transformation),Peter Fenwick (for the structured coding model, and many refinements),andAlistair Moffat, Radford Neal and Ian Witten (for the arithmeticcoder). I am much indebted for their help, support and advice.See the file ALGORITHMS in the source distribution for pointers tosources of documentation.Christian von Roques encouraged me to look for fastersorting algorithms, so as to speed up compression.Many people sent patches, helped with portability problems,lent machines, gave advice and were generally helpful.

This document was created byman2html,using the manual pages.