xmit | XMIT Session Layer Implementation | Application Framework library

If you are going to use PACKET_TX_RING, then you must play by its rules: that means you can't use the same buffer spot over and over: you must advance to the next buffer location within the ring buffer: build the first packet in slot 0, the second in slot 1, etc. (wrapping when you get to the end of the buffer).

But you're building every packet in slot 0 (i.e. at ps_header_start) but after sending the first packet, the kernel is expecting the next frame in the subsequent slot. It will never find the (second) packet you created in slot 0 until it has processed all the other slots. But it's looking at slot 1 and seeing that the tp_status in that slot hasn't been set to TP_STATUS_SEND_REQUEST yet so... nothing to do.

Note that your sendto call is not providing a buffer address to the kernel. That's because the kernel knows where the next packet will come from, it must be in the next slot in the ring buffer following the one you just sent.

This is why I suggested in your other question that you not use PACKET_TX_RING unless you really need it: it's more complicated to do correctly. It's much easier to just create your frame in a static buffer and call sendto(fd, buffer_address, buffer_len, ...). And if you are going to call sendto for each created frame, there is literally no advantage to using PACKET_TX_RING anyway.

If I follow the code correctly, you're redoing a ton of work for every IP address that doesn't need to be redone. Every time through the main loop you're:

• creating a new packet socket
• binding it
• setting up a tx packet ring buffer
• mmap'ing it
• sending a single packet
• unmapping
• closing the socket

That's a huge amount of work you're causing the system to do for one packet.

You should only create one packet socket at the beginning, set up the tx buffer and mmap once, and leave it open until the program is done. You can send any number of packets through the interface without closing/re-opening.

This is why your top time users are setsockopt, mmap, unmap, etc. All of those operations are heavy in the kernel.

Also, the point of PACKET_TX_RING is that you can set up a large buffer and create one packet after another within the buffer without making a send system call for each packet. By using the packet header's tp_status field you're telling the kernel that this frame is ready to be sent. You then advance your pointer within the ring buffer to the next available slot and build another packet. When you have no more packets to build (or you've filled the available space in the buffer [i.e. wrapped around to your oldest still-in-flight frame]), you can then make one send/sendto call to tell the kernel to go look at your buffer and (start) sending all those packets.

You can then start building more packets (being careful to ensure they are not still in use by the kernel -- through the tp_status field).

That said, if this were a project I were doing, I would simplify a lot - at least for the first pass: create a packet socket, bind it to the interface, build packets one at a time, and use send once per frame (i.e. not bothering with PACKET_TX_RING). If (and only if) performance requirements are so tight that it needs to send faster would I bother setting up and using the ring buffer. I doubt you'll need that. This should go a ton faster without the excess setsockopt and mmap calls.

Finally, a non-blocking socket is only useful if you have something else to do while you're waiting. In this case, if you have the socket set to be non-blocking and the packet can't be sent because the call would block, the send call will fail and if you don't do something about that (enqueue the packet somewhere, and retry later, say), the packet will be lost. In this program, I can't see any benefit whatsoever to using a non-blocking socket. If the socket blocks, it's because the device transmit queue is full. After that, there's no point in you continuing to produce packets to be sent, you won't be able send those packets either. Much simpler to just block at that point until the queue drains.

The ip4 checksum is only calculated over the ip header, if I get it correctly. So if you pass the total lenght of the whole packet to the checlsum calculation function, I would not be surprised, if you get a wrong checksum. I wonder though why it happend to work in the second program.

The ethernet, ip and tcp headers have to be populated. Here is what I came up with:

USB bulk endpoints implement a stream-based protocol, i.e. an endless stream of bytes. This is in contrast to a message-based protocol. So USB bulk endpoints have no concept of messages, message start or end. This also applies to USB CDC as it is based on bulk endpoints.

At the lower USB level, the stream of bytes is split into packets of at most 64 bytes. As per USB full-speed standard, packets cannot be larger than 64 bytes.

If the host sends small chunks of data that are more than 1ms apart, they will be sent and received in separate packets and it looks as if USB is a message-based protocol. However, for chunks of more than 64 bytes, they are split into smaller packets. And if small chunks are sent with less than 1ms in-between, the host will merge them into bigger packets.

Your design seems to require that data is grouped, e.g. the group of 720 bytes mentioned in the question. If this is a requirement, the grouping must be implemented, e.g. by first sending the size of the group and then the data.

Since larger groups are split into chunks of 64 bytes and the receive callback is called for every packet, the packets must be joined until the full group is available.

Also note a few problems in your current code (see usbd_cdc_if.c, line 264):

I finally managed to do what I want.

My team wished to use Ansible for the formatting, so I had to improvise a bit.

I used ntc-ansible for that.

With the help of members of the NTC Slack, I finally got it working. Here's what I came up with:

A functionality that is very poorly documented on the TextFSM repo is that you can, in an index file, combine two templates that share a common "Key" attribute.

So I created two templates: