
	      'ZIPLIB' COMPRESSED DATA FORMAT SPECIFICATION
				draft #3
			    March 7, 1995

		    Copyright (C) 1995 L. Peter Deutsch and Jean-loup Gailly

	Permission is granted to copy and distribute this document for
	any purpose and without charge, provided that it is copied as a
	whole (including the copyright notice and this notice) and with
	no changes.

1. Introduction

1.1 Purpose

The purpose of this specification is to define a lossless compressed data
format that:

	(a) Is independent of CPU type, operating system, file system, and
character set, and hence can be used for interchange;

	(b) Can be produced or consumed, even for an arbitrarily long
sequentially presented input data stream, using only an a priori bounded
amount of intermediate storage, and hence can be used in data communications
or similar structures such as Unix filters;

	(c) Can use a number of different compression methods;

	(d) Can be implemented readily in a manner not covered by patents,
and hence can be practiced freely.

The data format defined by this specification does not attempt to
allow random access to compressed data.


1.2 Intended audience

This specification is intended for use by implementors of software to
compress data into ziplib format and/or decompress data from ziplib format.

The text of the specification assumes a basic background in programming at
the level of bits and other primitive data representations.

1.3 Scope

The specification specifies a compressed data format, to be used for
in-memory compression of a sequence of arbitrary octets.

1.4 Compliance

Unless otherwise indicated below, a compliant decompressor must be able to
accept and decompress any data set that conforms to all the specifications
presented here; a compliant compressor must produce data sets that conform
to all the specifications presented here.

1.4 Related standards

None.

1.5 Other related publications

Deutsch, L.P.,"'Gzip' Compressed Data Format Specification".  Drafts
being circulated.

Deutsch, L.P.,"'Deflate' Compressed Data Format Specification".  Drafts
being circulated.

Thomas Boutell, "PNG (Portable Network Graphics) specification".

Fletcher, J. G., "An Arithmetic Checksum for Serial Transmissions," IEEE  
Transactions on Communications, Vol. COM-30, No. 1, January 1982, pp.  
247-252.

ITU-T Recommendation X.224, Annex D, "Checksum Algorithms," November,  
1993, pp. 144, 145. (Available from gopher://info.itu.ch). ITU-T X.244 is
also the same as ISO 8073.


1.6 Definitions of terms and conventions used

octet: 8 bits stored or transmitted as a unit (same as a byte on most
machines).  See section 3.1 below for the numbering of bits within an
octet.


2. Detailed specification.

2.1 Overall conventions.

In the diagrams below, a box like this:

	+---+
	|   | <-- the vertical bars might be missing
	+---+

represents one octet; a box like this:

	+==============+
	|              |
	+==============+

represents a variable number of octets.

Octets stored within a computer do not have a 'bit order', since they are
always treated as a unit.  However, an octet considered as an integer
between 0 and 255 does have a most- and least-significant bit, and since we
write numbers with the most-significant digit on the left, we also write
octets with the most-significant bit on the left.  In the diagrams below, we
number the bits of an octet so that bit 0 is the least-significant bit,
i.e., the bits are numbered:

	+--------+
	|76543210|
	+--------+

Within a computer, a number may occupy multiple octets.  All multi-octet
numbers in the format described here are stored with the MOST-significant
octet first (at the lower memory address).  For example, the decimal number
520 is stored as:

	    0	     1
	+--------+--------+
	|00000010|00001000|
	+--------+--------+
	 ^	  ^
	 |	  |
	 |	  + less significant octet = 8
	 + more significant octet = 2 x 256

2.2 Data format.

The data have the following structure:

	  0   1  
	+---+---+=====================+---+---+---+---+
	|CMF|FLG|...compressed data...|    ADLER32    |
	+---+---+=====================+---+---+---+---+

CMF (Compression Method and flags)
        This octet is divided into a 4-bit compression method and a 4-bit
        information field depending on the compression method.
                bits 0 to 3  CM     Compression method
                bits 4 to 7  CINFO  Compression info
  CM (Compression method)
	This identifies the compression method used in the file. CM = 8
    denotes the 'deflate' compression method with a window size up to 32K.
    This is the method used by gzip; it is documented elsewhere.
  CINFO (Compression info)
        For CM = 8, CINFO is the base-2 logarithm of the LZ77 window size,
        minus eight (CINFO=7 indicates a 32K window size). Values of CINFO
        above 7 are not allowed in this version of the specification.
        CINFO is not defined in this specification for CM not equal to 8.

FLG (FLaGs)
	This flag octet is divided as follows:

		bits 0 to 4  FCHECK  (check bits for CMF and FLG)
		bit  5       reserved, must be zero
		bits 6 to 7  FLEVEL  (compression level)

  The FCHECK value must be such that CMF and FLG, when viewed as a 16-bit
  unsigned integer stored in MSB order (CMF*256 + FLG), is a multiple of 31.

  FLEVEL (Compression level)
	These flags are available for use by specific compression methods.
    The 'deflate' method (CM = 8) sets these flags as follows:
	0 - compressor used fastest algorithm
	1 - compressor used fast algorithm
	2 - compressor used default algorithm
        3 - compressor used maximum compression, slowest algorithm
     The information in FLEVEL is not needed for decompression; it is there
     to indicate if recompression might be worthwhile.

ADLER32 (Adler-32 cheksum)
	This contains a checksum value of the uncompressed data
computed according to Adler-32 algorithm. This algorithm is a 32-bit
extension and improvement of the Fletcher algorithm, used in the
ITU-T X.224 / ISO 8073 standard.

Adler-32 is composed of two sums accumulated per octet: s1 is the sum
of all octets, s2 is the sum of all s1 values. Both sums are done
modulo 65521. s1 is initialized to 1, s2 to zero.  The Adler-32
checksum is stored as s2*65536 + s1 in most-significant-octet first
(network) order.


3. Appendix: Rationale

The Adler-32 algorithm is much faster than the CRC32 algorithm yet
still provides an extremely low probability of undetected errors.

The modulo on unsigned long accumulators can be delayed for 5552 octets, so
the modulo operation time is negligible.  If the octets are a, b, c, the
second sum is 3a + 2b + c + 3, and so is position and order sensitive,
unlike the first sum, which is just a checksum.  That 65521 is prime is
important to avoid a possible large class of two-octet errors that leave
the check unchanged.  (The Fletcher checksum uses 255, which is not prime
and which also makes the Fletcher check insensitive to single octet changes
0 <-> 255.)

The sum s1 is initialized to 1 instead of zero to make the length of the
sequence part of s2, so that the length does not have to be checked
separately. (Any sequence of zeroes has a Fletcher checksum of zero.)


The following C code computes the Adler-32 checksum of a data buffer:

#define BASE 65521 /* largest prime smaller than 65536 */
#define NMAX 5552
/* NMAX is the largest n such that 255n(n+1)/2 + (n+1)(BASE-1) <= 2^32-1 */

unsigned long adler32(unsigned char *b, int n)
{
  unsigned char *p = b;
  unsigned long s1 = 1;
  unsigned long s2 = 0;
  int k;

  while (n > 0) {
    k = n < NMAX ? n : NMAX;
    n -= k;
    do {                        /* could unroll loop some for speed */
      s1 += *p++;
      s2 += s1;
    } while (--k);
    s1 %= BASE;
    s2 %= BASE;
  }
  return (s2 << 16) | s1;
}
