pyembroidery | libembroidery/EmbroideryFormats converted to python | Machine Learning library

ArchiveLib compression was a sublicensed product by Robert Jung who did ARJ compression. The same scheme is used in ARJ as well as ArchiveLib and the Patent protection has since expired.

It sends compressed huffman tables by which are built by reading n huffman_code_length. These are checked as to whether they solved for a balanced huffman tree. These are sorted from longest code word to shortest code word with tied entries going to the lowest value.

So if the lengths of the values are 4, 8, 2, 2. It gives these 8, 4, 2, 2 With codewords 0, 10, 110, 111. The order these were given reflects their positioning. So the first entry gets 10, second 0, and third 110 and forth 111. Since these two tied, they were applied in the order received.

Blocks are read by 16 bytes to convey the number of symbols. This is followed by the character length table This is huffman encoded. The character length table is then used to build the character table. Using the huffman table to build the character huffman table. Then a third distance table is send which gives the values for the lookback within the window for the read.

Character Length Table.

5 bits of count data. This can convey values 0 to 32 entries. If zero another 5 bit value is read which is used for all values.

For each value in the count we read a variable length. With a special value at index 3 (4th value given) is prefixed by 2 bits that skip forward in the index by 0 to 3 values. These skipped values are treated as part of the count. These first three entries are special command codes in the character table.

This will then provide up to 32 different values lengths the huffman code lengths can be.

Character Table.

9 bits of count. This means it can provide values from 0 to 511. If 0 then all values are the 9 bits directly following that.

The number of entries are the values that actually fill the table. So if 4 then we will have 4 different non-zero entries in the 511 entry table.

The data encoded is how to skip or fill the relevant data provided. 0 means we set one zero in the table sequence. A 1 means we read 4 bits and add 3 to that value and zero out that many table entries. A 2 means we read 9 bits and add 20 to that value and zero out that many table entries. For any value greater than that, we subtract 2 and simply put it that length in the table.

The result is a list of lengths for all the required codes for the, up to, 511 elements (any entries that didn't get a value are 0). This is then sent to the huffman table building algorithm to be checked for validity and have the table built. This assigns huffman codes to tokens from 0 to 511.

Distance Huffman

5 bits of count. If zero subsequent 5 digits is given for all values.

Each entry reads a variable length data value. This results in 0-32 values of some length. These are then used to build a relevant huffman table.

This table will have up to 32 different integer values defined by a huffman table.

Variable Length Data.

3 bits of value. If not 7 return that value. If 7 read between 0 and 13 additional bytes until a non-one value is found. So if a value is less than 7 that value is used. If greater than 7, each additional 1 refers to a value 1 greater. So 11110 is 8. 111110 is 9. Up to 13 bits. This gives values from 0 to 20. In between 3 and 16 bits.

Decompression.

Actual decompression means reading the block and tables. Then the compressed stream using the character table and, when needed, the distance table. Each value is checked against the character table to provide the token. If the value is 0-255 that is a literal. If the value is 510 it denotes an end-command. If the value is greater than 255 it corresponds to a pointer back into the window. Values greater than 255 correspond to a string of (v - 253) characters. Which is to say a value of 256 means we are reading 3 characters from the window. 257 means 4 characters. Up to 509 which means we're reading 255 characters from the window. If we are reading a from a lookup the next value on the stream is a backwards distance read using the Distance Huffman table.

Compression.

Any compression scheme that fits the decompression scheme will work. How it finds encodes things is of little concern. The Patent on ARJ was to simply the manner of how to compress things rapidly, this is not much of a concern. However an interesting methodology would be to not compress things at all but simply force a null-op through the header.

An example of writing null-op compression.

16 bits of block size. If we need more than 32k elements we'll have to write another block.

• Writing Character Length Huffman. 5 bytes: 00000, we have 0 entries. 1 length. They are all 8. 5 bytes: 01010, all values are 10, meaning length 8.

The Character Length huffman only returns 10.

• Writing Character Huffman. 9 bits: 100000000, 256 entries.

All entries will query the Character Length huffman and get a value 10, which is 8.

The Character Huffman table will have 256, 8 bit entries. This will build a literals table where every byte exactly equals itself. Since there are 256 tied entries they are taken in lowest element first.

Distance Huffman. (we are never using this). 5 bits: 00000 No entries. 5 bit: 00000 Any value doesn't matter.

This makes writing our literals table take 29 bits. However, for this method to be highly useful we want it to exactly divide 8 bits. We can do this by changing our distance huffman, which we never use since all values are literals.

5 bit: 00001 1 distance huffman. 3 bit: 111 we have a variable length value 7 or more. 5 bit: 11110 we add 4 to this 7 for a length of 11. But really we're padding.

(10) + (9) + (5 + 3 + 5) = 32 bits.

0b00000010101000000000000111111110 This is storable in a single integer: 0x02A001FE

So to compress we can simply give the size of the block, then 0x02A001FE and our uncompressed data. And it will count as compressed. Since I can't actually recommend using a mostly dead compression scheme.