NALU | Keras Implementation of : Neural Arithmetic Logic Units | Machine Learning library

my goal is to encode the main framebuffer of my Windows machine using nvenc and stream its content to my iPad using the VideoToolbox API

The code I use to encode the h264 stream is basically a copy/paste of https://github.com/NVIDIA/video-sdk-samples/tree/master/nvEncDXGIOutputDuplicationSample the only change is that instead of writing to a file, I do send the data

For the decoding I do use https://github.com/zerdzhong/SwfitH264Demo/blob/master/SwiftH264/ViewController.swift#L71

The encoding work perfectly when I write all the contents to a file, I am able to use a h264->mp4 online converter without issue, the problem is that the decoder gives me the error kVTVideoDecoderBadDataErr in the function decompressionSessionDecodeFrameCallback

So for what I tried:

• Firsly using an h264 analyzer I found that the frame order are: 7/8/5/5/5/5/1...
• I found that nvenc does encode the frames 7/8/5/5/5/5 in only one packet
• I did try to separate this packet into multiple ones using the sequence (0x00 0x00 0x00 0x01), it gave me the frames 7/8/5 separately
• As you can see I only got one 5 frame which is around 100KB, the H264 analyzer said that there are four 5 frames (which are something like 40KB, 20KB, 30KB, 10KB)
• Using a hex file viewer I saw that the sequence separating those 5 frames were (0x00 0x00 0x01) instead, tried to also separate them but I got the exact same VideoToolbox error while decompressing

here is the code I use to separate and send the frames: The protocol is simply PACKET_SIZE->PACKET_DATA The swift code is able to read the NALU types so I am confident that this is not the issue

I'm attempting to extract raw image data for each I-frame from an MPEG-2 Transport Stream with a H.264 annex B codec. This video contains I-frames on every 2 second interval. I've read that an I-frame can be found in after an NALu start code with a type of 5 (e.g. Coded slice of an IDR picture). The byte payload of these NALu's contains all the necessary data to construct a full frame. Albeit to, my understanding, in a H.264 encoded format.

I would like to build a solution to extract these I-frame from an incoming byte stream, by finding NALu's that contain I-frames, saving the payload and decoding the payload into some ubiquitous raw image format to access pixel data etc.

Note: I would like to avoid using filesystem dependency binaries like ffmpeg if possible and more importantly if feasible!

So far I have build an PoC in rust to find the byte offset, and byte size of I-frames:

there are a h264 rtp stream passing through server, I catch every rtp packets and extract every nalu by getting rid of rtp header, and then write every nalu to file record.h264 with adding a h264 prefix start code 0x00000001, but file record.h264 cannot be played by ffplay and cannot be parsed by ffprobe, where is wrong?

here is my record file:https://github.com/sshsu/record_h264_file

Suppose I am receiving packets containing H264 encoded NAL Units, and each NAL has correct START_CODE (0x00 00 00 01).

But in some NALUs, the later Bytes of the NAL may have been corrupted. Is there any standard way to validate whether the information represented by a NALU is correct or not?

And if it can be validated, is there any standard way to correct it according to the H264 standards ?

Further Clarification:

The problem to specify is somewhat like this ->

Server sends a x264 NALU like 0x 00000001abcdefgh and, when I receive the packet, it contains : 0x 00000001abcmnopdefgh.

This leads to failure of decompression , hence I was looking for a way to eliminate these bad data bytes.

I recently got HEVC / H.265 decoding working using VideoToolbox api in iOS. The incoming RTP stream came from FFMPEG using the x265 codec in libx265.

After much tinkering, I created my CMFormatDescription using the incoming VPS, SPS and PPS nalus - then I waited for a CRA_NUT nalu (21) keyframe. Once that came through, I could decode the next wave of video packets received. Great!

However I want to now accept an RTP stream encoded by Nvidia's Video Coding SDK. The difference is though that instead of a CRA_NUT nalu arriving after the sequence parameters - I am now receiving an IDR_W_RADL (19) which VideoToolbox doesn't seem to like - as a result i get kVTVideoDecoderBadDataErr from my decompression session's callback.

Since the documentation is quite sparse for VideoToolbox - it's quite difficult to debug. Does VideoToolBox always expect a CRA_NUT nalu as a keyframe? Or is there any way to make it accept an IDR_W_RADL keyframe? Is there a standard sequence VideoToolbox expects?

Conversely, is there any way to configure the nvidia video codec sdk to return a CRA_NUT keyframe? I've looked through the api but can't find anything - i always seem to get IDR_W_RADL along with the sequence parameters.

I'm trying to make a FLV muxer. And I'm started to test on a h264 stream I capture from my camera (c920). After encoding the .flv file, it doesn't play correctly.

First I tried to find NALs in h264 searching for the pattern 0x00 0x00 0x00 0x01... but I found in the internet that there are two patterns to find NALs... So I implemented the searching for 0x00 0x00 0x01 and the other...

In firsts tests I found that are few NALs starting with four bytes, but after changing the code to search the 3 bytes pattern I found a lot of NALs...

The references I found in the internet shows code using few NAL types to encapsulate the FLV file, but after the change to detect 3 bytes, the program found a lot of NALs and I don't know how to stream them...

Some code :-)

I'm implementing in C++, so I have the classes.

The FLVWritter

MP4 file (actually it's HLS stream that uses MP4 format) contains H.264 stream in AVC format: [NALU length][NALU data]. H.264 decoder requires Annex B format: [startcode][NALU data]. In general startcode emulation can happen in AVC format (e.g. 0x00000001 can occur inside encoded data). Does this mean that MP4 file format also can contain H.264 NALUs with startcode emulation? Or should emulation prevention be applied before packaging into MP4?

MediaMuxer has been driving me mad for two days & nights now :-(

The situation: I receive a H264 encoded 1280x720 video stream via UDP. The h264 stream contains NALU 1 - slice and NALU 5 - keyframe (5 is always preceded by NALU 7 - SPS and NALU 8 - PPS). This stream appears to be stable 30fps with at least one NALU 5 keyframe per second. Bitrate is variable but less than 4Mbps. MediaCodec sucessfully decodes the stream and renders it in a surface view so that part works well.

But now I need to save the H.264 into a local mpg file. I set up a MediaMuxer with all MediaFormat information that I have, and feed it with the sample data from the stream. Each sample contains one frame (NALU 1 or 5), and the first data sent to MediaMuxer is a keyframe (NALU 5). The presentation time is calculated based on framenumber and framerate. All involved methods are called from the same thread.

But the mpg file is never created. As you can see in the output below the data in the ByteBuffer does start with NALU headers followed by varying size of data. MediaMuxer seems to "see" frames in the data as it counts the frames. So what is wrong here?

Minimum API is 21, and I have tested with a Samsung Galaxy S4 running stock Android 5 and a couple of devices running Lineageos Oreo and Nougat.

Here is the code to setup the MediaMuxer:

Can anyone tell me or point me to a section of the specification(s) that clearly demonstrates how from an elementary stream with a series of NALUs how these should be written into a ISO BMFF mdat?

I can see looking at samples and other code that I should have something like: AUD, SPS, PPS, SEI, VideoSlice, AUD etc etc

Things that are not entirely clear to me:

1. If the SPS and PPS are also stored out of band in the AVCC are they required in the mdat?
2. If they are required in the mdat when/where should they be written? e.g. just prior to an IDR?
3. What is the requirement for AUDs?
4. If I am generating sample sizes for the trun is the calcuation for this? In the example I am working to recreate the first sample in the trun has a size of 22817 however if I look at the first sample in the mdat the NALU size prefix is 22678. The value in the trun appears to be the size of all the NALUs + sizes up to and including the first sample (see my example below)

>