EliasFanoCompression | succinct compression of sorted integers | Compression library

Good Day!

I would like ask for your help on decompressing String back to its original data.

Here's the document that was sent to me by the provider.

Data description

First part describes the threshold data.

All data are managed as Little Endian IEEE 754 single precision floating numbers. Their binary representation are (represented in hexadecimal data) :

Compressed data (zip) Threshold binary data are compressed using the ‘deflate’ algorithm. Each compression result is given here (represented in hexadecimal data) :

Thresholds: $63 00 03 05 47 24 DA 81 81 A1 C1 9E 81 61 01 98 06 00

Encoded data (base64) Threshold compressed data are encoded in ‘base64’ to be transmitted as ASCII characters. Each conversion results is given here (represented in hexadecimal data) :

Thresholds: $59 77 41 44 42 55 63 6B 32 6F 47 42 6F 63 47 65 67 57 45 42 6D 41 59 41

Here is the output frame (Manufacturer frame content) The thresholds data are then sent using their corresponding ASCII character Here is the resulting Histogram ASTM frame sent :

YwADBUck2oGBocGegWEBmAYA

As explained in above details, what I want to do is backwards.

The packets that we received is

YwADBUck2oGBocGegWEBmAYA

then from there convert it to Hex value Base64 which is the output is.

Thresholds: $59 77 41 44 42 55 63 6B 32 6F 47 42 6F 63 47 65 67 57 45 42 6D 41 59 41

This first part was already been implemented using this line of codes.

Is there a way during the ffmpeg compression process to determine over various intervals the exact filesize that a video is at?

Such as a method to get current filesize during the process to use when comparing against the videos original filesize.

For example, a potential video being transcoded takes 5 minutes, but during the process, a function will check the file size on intervals of 100 frames or every 5 seconds to ensure that the filesize hasn't exceeded the original. If it has, it will kill the process with command.kill('SIGSTOP');

Is the Shannon-Fano coding as described in Fano's paper The Transmission of Information (1952) really ambiguous?

3 papers
Claude E. Shannon published his famous paper A Mathematical Theory of Communication in July 1948. In this paper he invented the term bit as we know it today and he also defined what we call Shannon entropy today. And he also proposed an entropy based data compression algorithm in this paper. But Shannon's algorithm was so weak, that under certain circumstances the "compressed" messages could be even longer than in fix length coding. A few month later (March 1949) Robert M. Fano published an improved version of Shannons algorithm in the paper The Transmission of Information. 3 years after Fano (in September 1952) his student David A. Huffman published an even better version in his paper A Method for the Construction of Minimum-Redundancy Codes. Hoffman Coding is more efficient than its two predecessors and it is still used today. But my question is about the algorithm published by Fano which usually is called Shannon-Fano-Coding.

The algorithm
This description is based on the description from Wikipedia. Sorry, I did not fully read Fano's paper. I only browsed through it. It is 37 pages long and I really tried hard to find a passage where he talks about the topic of my question, but I could not find it. So, here is how Shannon-Fano encoding works:

1. Count how often each character appears in the message.
2. Sort all characters by frequency, characters with highest frequency on top of the list
3. Divide the list into two parts, such that the sums of frequencies in both parts are as equal as possible. Add the bit 0 to one part and the bit 1 to the other part.
4. Repeat step 3 on each part that contains 2 or more characters until all parts consist of only 1 character.
5. Concatenate all bits from all rounds. This is the Shannon-Fano-code of that character.

An example
Let's execute this on a really tiny example (I think it's the smallest message where the problem appears). Here is the message to encode:

Final Update: Turns out I didn't need Binary writer. I could just copy memory streams from one archive to another.

I'm re-writing a PowerShell script which works with archives. I'm using two functions from here

Expand-Archive without Importing and Exporting files

and can successfully read and write files to the archive. I've posted the whole program just in case it makes things clearer for someone to help me.

However, there are three issues (besides the fact that I don't really know what I'm doing).

1.) Most files have this error on when trying to run Add-ZipEntry -ZipFilePath ($OriginalArchivePath + $PartFileDirectoryName) -EntryPath $entry.FullName -Content $fileBytes}

Cannot convert value "507" to type "System.Byte". Error: "Value was either too large or too small for an unsigned byte." (replace 507 with whatever number from the byte array is there)

2.) When it reads a file and adds it to the zip archive (*.imscc) it adds a character "a" to the beginning of the file contents.

3.) The only file it doesn't error on are text files, when I really want it to handle any file

Thank you for any assistance!

Update: I've tried using System.IO.BinaryWriter, with the same errors.

I have been trying to make a python script to zip a file with the zipfile module. Although the text file is made into a zip file, It doesn't seem to be compressing it; testtext.txt is 1024KB whilst testtext.zip (The code's creation) is also equal to 1024KB. However, if I compress testtext.txt manually in File Explorer, the resulting zip file is compressed (To 2KB, specifically). How, if possible, can I combat this logical error?

Below is the script that I have used to (unsuccessfully) zip a text file.

I am learning F# and Deedle. I am trying to extract the contents of this TGZ File using SharpZipLib. I downloaded the TGZ to my local drive. I think I am close because out1 works, but out2 errs. I am sure the code could be written better with pipe forwarding or composition, but it first needs to work. Does anyone have any ideas?

I've been working on a little side project of listing files compressed in nested zip files. I've cooked up a script that does just that, but only if the depth of zip files is known. In in example below the zip file has additional zips in it and then anthoer in one of them.

I would like to include -co options to compress output raster using gdalwarp from gdalUtilities in R.

I have tried some options (commented in the code), but I have not been successful in generating the compressed raster.

I have managed to convert zip files to gzip, using AWS lambda's local storage /tmp. The problem is that this storage maxes out at 500Mb.