Transcoder | Docker container to transcode videos | Continuous Deployment library

Turns out this is possible through the use of an undocumented flag, FLAG_RTSP_CUSTOM_IO. I've detailed how to do that here: https://blog.kevmo314.com/custom-rtp-io-with-ffmpeg.html

I'm not sure what you're using to code, but in order to set it in Android Studio, open the manifest of your project and under the "activity" section, put android:exported="true"(or false if that is what you prefer). I have attached an example.

The streaming options need to be passed in an array, not a single string with the separator as you have used.

This is a fragment of what you have:

You are getting high frame rate because you have failed to set packet duration.

• Set the time_base to higher resolution (like 1/60000) as described here:

I would do this using the transcoder API in GCP. It supports mp4 input and hls output. See supported formats

Note that videos must be greater than 5 seconds in length. If they can't be 5 seconds in length, maybe avoid this API and using a tool on AppEngine.

A rough flow of events to accomplish this could be something like:

1. Upload MP4 to bucket
2. pubsub is triggered from the upload event
3. This can trigger a cloud function that can create a new Transcoder job
4. The job status updates a pubsub topic
5. Pubsub topic triggers a cloud function when the status indicates the job is done
6. Cloud function deletes the original mp4 from the bucket

So, after almost two entire days of trial and error plus digging into the source code, I finally found a solution! To make it short: unauthenticated access and MediaConvert will not work!

The entire problem is Cognito which does not allow access to MediaConvert operations on unauthenticated access. Here is the access list.

Since I am using Auth0 for my user authentication I was simply following this guide and basically all my problems were gone! To attach the token I was using

while I'm not 100% sure this is the problem, I believe the Paperclip library is deprecated and it does not work with ruby 3. If you use the latest ruby 2 version is should work for you. However, I would recommend not depending on paperclip for new projects.

There are some defaults not listed in the documentation. Try adding the following to your config and see whether it works

I was hoping that Elastic Transcoder would provide the transcoded files ETag in the job response

Unfortunately, it doesn't.

As of the current AWS Python Boto3 1.18.60 SDK (or any other SDK including the REST API), the entity tag(s) for the output object(s) isn't returned anywhere in the job response object.

This is most likely because an entity tag represents a specific version of the object and is mostly used for efficient caching invalidation.

Elastic Transcoder jobs do not produce multiple versions of the same output and as such, why would it return the ETag value? If someone requires the ETag, they can get this from the S3 object.

Another question is what happens if there is multi-part output for large inputs? What does the SDK return? A list of ETag header values? You have multiple parts, but you don't have multiple versions.

This implementation would go against the RFC 7232 specification for the ETag header:

An entity-tag is an opaque validator for differentiating between multiple representations of the same resource, regardless of whether those multiple representations are due to resource state changes over time, content negotiation resulting in multiple representations being valid at the same time, or both.

Your actual problem, in this case, is that you want the MD5 hash of the file(s) even if they are multi-part.

Now your code will work for getting the MD5 hash for a single file, but if they are multi-part, they don't hash the multipart uploads as you would expect. Instead of calculating the hash of the entire file, Amazon calculates the hash of each part and then combines that into a single hash set as the ETag header.

This makes a lot of sense: they calculate the hashes of each part as they receive it. After all the parts have been transferred, they combine the hashes instead of trying to calculate a final MD5 hash by reading a file that could possibly be up to the AWS object size limit - 5 terabytes. Try generating MD5 hashes for everyone's files when you're at Amazon's scale & you'll find that their way is quite clever :)

This is probably why the S3 API Reference says the below:

The ETag may or may not be an MD5 digest of the object data.

Objects created by either the Multipart Upload or Part Copy operation have ETags that are not MD5 digests, regardless of the method of encryption.

It is an MD5 hash, but when it's a multi-part upload it isn't so the above is technically correct.

To calculate the MD5 hash correctly for multi-part uploads, try checking out this great answer in response to "What is the algorithm to compute the Amazon-S3 Etag for a file larger than 5GB?".

To summarise, you, unfortunately, don't have the MD5 digest hash of the object(s) in the response back from Amazon Elastic Transcoder - you'll have to do this heavy lifting yourself & it can take long if you have huge files.

There's no workaround or quicker solution - you have the quickest solution as you're already getting the ETag value from the HEAD in the most efficient way.

I would perhaps recommend trying to parallelise the HTTP HEAD requests made to get the objects' metadata (s3_cli.head_object(...)) before trying to determine the final MD5 digest of the file(s).

That will definitely speed things up if you have 500 files for example - don't make the API requests separately.

You’ll definitely be able to send them in parallel, so offload the time you’d spend between requests all onto Amazon’s infrastructure.

Get the responses & then process together.