HuffMan- | huffman compression function | Compression library

The zlib library supports dictionaries with the zlib (not gzip) format. See deflateSetDictionary() and inflateSetDictionary().

There is nothing special about the construction of a dictionary. All it is is 32K bytes of strings that you believe will occur often in the data you are compressing. You should put the most common strings at the end of the 32K.

It is generating the codes correctly, but then printing them incorrectly. It is leaving off the leading zero bits of the codes that have them. They should have prepended the necessary zero bits after converting the number to a string of digits.

If you replace the line that prints the code with this:

I haven't weeded through the code enough to know why yet, but huffmandict is not ignoring zero-probability symbols the way it claims to. Nor have I been able to find a bug report on Savannah, but again I haven't searched thoroughly.

A workaround is to limit the symbol list and their probabilities to only the symbols that actually occur. Using containers.Map would be ideal, but in Octave you can do that with a couple of the outputs from unique:

Assuming that your Huffman tree is valid (meaning we can ignore :frequency), and that 0 means 'left' and 1 means 'right':

Yes, there are several ways to generate prefix-free codes dynamically.

As you suggested, it would be conceptually simple to start with some default frequency, track the frequencies of the letters used so far, and for every letter decoded, increment that letter's count and then re-build a Huffman tree from all the counts. (potentially completely changing the tree after each letter). That would require a lot of work for each letter and be very slow -- and yet there are a couple of adaptive Huffman coding algorithms that effectively do the same thing -- using clever algorithms that do much less work, and so are faster.

Many other data compression algorithms also generate prefix-free codes dynamically much faster than any adaptive Huffman algorithm, at a small sacrifice of compression -- such as Polar codes or Engel coding or universal codes such as Elias delta coding.

The arithmetic coding data compression algorithm is technically not a prefix-free code, but typically gives slightly better compression (but runs slower) than either static Huffman coding or adaptive Huffman coding. Arithmetic coding is generally implemented adaptively, tracking the frequencies of all the letters used so far. (Many arithmetic coding implementations track even more context -- if the previous letter was a "t", it remembers that the most-frequent letter in this context is "h" and exactly how frequent it was, etc., giving even better compression).

1. We don't know for sure why Phil Katz chose 15, but it was likely to facilitate a fast implementation in a 16-bit processor.

2. No, zlib will not fail. It happens all the time. The zlib implementation applies the normal Huffman algorithm, after which if the longest code is longer than 15 bits, it proceeds to modify the codes to force them all to 15 bits or less.

Note that your example resulting in a 256-bit long code would require a set of 2²⁵⁶ ~= 10⁷⁷ symbols in order to arrive at those frequencies. I don't think you have enough memory for that.

In any case, zlib normally limits a deflate block to 16384 symbols. For that number, the maximum Huffman code length is 19. That comes from a Fibonacci sequence of probabilities, not your powers of two. (Left as an exercise for the reader.)

How about:

Huffman code works by laying out data in a tree. If you have a binary tree, you can associate every leaf to a code by saying that left child corresponds to a bit at 0 and right child to a 1. The path that leads from the root to a leaf corresponds to a code in a not ambiguous way.

This works for any tree and the prefix property is based on the fact that a leaf is terminal. Hence, you cannot go to leaf (have a code) by passing though another leaf (by having another code be a prefix).

The basic idea of Huffman coding is that you can build trees in such a way that the depth of every node is correlated with the probability of appearance of the node (codes more likely to happen will be closer the root).

There are several algorithms to build such a tree. For instance, assume you have a set of items you want to code, say a..f. You must know the probabilities of appearance every item, thanks to either a model of the source or an analysis of the actual values (for instance by analysing the file to code).

Then you can:

1. sort the items by probability
2. pickup the two items with the lowest probability
3. remove these items, group them in a new compound node and assign one item to left child (code 0) and the other to right child (code 1).
4. The probability of the compound node is the sum of individual probabilities and insert this new node in the sorted item list.
5. goto 2 while the number of items is >1

For the previous tree, it may correspond to a set of probabilities

a (0.5) b (0.2) c (0.1) d (0.05) e (0.05) f (0.1)

Then you pick items with the lowest probability (d and e), group them in a compound node (de) and get the new list

a (0.5) b (0.2) c (0.1) (de) (0.1) f (0.1)

And the successive item lists can be

a (0.5) b (0.2) c(de) (0.2) f (0.1)

a (0.5) b (0.2) (c(de))f (0.3)

a (0.5) b((c(de))f) (0.5)

a(b(((c(de))f)) 1.0

So the prefix property is insured by construction.

Don't use a String to store the binary result, use a java.util.BitSet.

It does exactly what you want, allowing you to set individual bits by index position.

When you are ready to extract the value in binary you can use toByteArray().