dpdk | comprehensive rust binding for DPDK allowing high speed | Networking library

Note: I highly recommend reading about dpdk testpmd as it covered all your questions in detail. As per the StackOverflow guideline multiple sub-questions make answering difficult. Please use well-formatted and formed question for better reach and answers.

@Alexcured since you have mentioned you know how to run 2 separate instances of dpdk-testpmd. I will only recommend heavily reading up on dpdk-testpmd URL which has answers to most of your questions too.

The assumption is made, both the PCIe NIC are working properly and interconnect between 2 are tested with either arping or ping (Kernel Driver). After binding both PCIe devices to DPDK supported drivers, one should use options for DPDK 20.11.1 such as

• use file-prefix option as unique names
• set socket-memory to fetch memory from the desired NUMA-SOCKET
• set socket-limit to prevent ballooning of huge page mmap
• use w|b option to whitelist|blacklist PCIe devices (0000:3b:00.0 and 0000:3b:00.1)
• Since it is separate physical devices ensure there physical cable connection between 2 PCIe ports.

[Q.1] How to set the MAC address of instance 2's port in instance 1?

You need to modify the makefile inorder to adapt C++:
You need to change CFLAGS to CPPFLAGS See below reference example:

[answer based on the live debug] there is no issue on the DPDK side with either e1000e or vmxnet3 device. Based on the current debug it based on switch on vsphere

Debugging with kernel driver vmnext3, shows

1. VM-1 receives and transmit packets
2. VM-2 receives and transmit packets
3. there is a physical interface also connected which forwards packets.
4. set the IP address of VM-1 to '12.12.12.1' and IP address of VM-2 to '12.12.12.2'
5. Run iperf3 between VM-1 and VM-2, it fails.
6. debugging with tcpdump showcases that VM-1 is not receiving packets from the switch from VM-2. While packets from VM-1 are received.

The same is behaviour with DPDK l2fwd/testpmd/skeleton and even sample application. Hence DPDK 18.11.11 LTS with vmxnet3 has no issues in sending or receiving packets.

Assuming you have not used a custom path for DPDK install and used meson for building dpdk 19.08, there are 2 ways to do it

1. ignore the libraries installed from DPDK 19.08, simply execute sudo ninja install. This overrides all libraries, drivers and include.
2. if want to do it cleanly, go to DPDK 19.08 build folder, execute sudo ninja uninstall. This will remove your previous installation. Then build Dpdk 20.11.1 LTS and do sudo ninja install

A good practice is always creating a custom path by -Dprefix=/opt/dpdk-install/.

note: If you have used Makefile for DPDK-19.08, one can always remove the dpdk folder (if it had been created with make config -T=[target] -O=[output folder]

[EDIT-1] update from live debug

Target OS: CENTOS 7 DPDK version: 20.11.1

There are multiple reasons which were causing error

1. Installing Mellanox libverbs affects the cflags and ldflags for dpdk libdpdk.pc
2. DPDK 21.11.1 meson build (meson -Dexamples=l2fwd build; ninja -C install) does not match l2fwd Makefile option of make static
3. using pkg-config from the installed target was also incorrect.

To fix the error we edited /usr/local/lib64/pkgconfig/libdpdk.pc to include the static libraries under -Wl,--whole-archive and -Wl,--no-whole-archive

it works

XDP (Express Data Path) supported in Linux is for RX side, and there were patches for TX but not integrated. Based on the current update XDP-eBPF is on Physical NIC. So all RX packet on physical NIC is processed.

But as per the question shared packets from the guest should arrive/be injected to the NIC driver ( XDP working in native mode ) of the host.. If one needs to run the logic for traffic coming from GUEST OS, XDP has to be loaded to emulator or TAP or Bridge interface. This will allow to redirect packets based on Kernel NIC id to Physcial NIC.

[Answer is based on the live debug and configuration settings done to improve performance]

Factors that were affecting performance for Kernel and DPDK interfaces were

1. Host system was not using CPU which were isolated for VM
2. KVM-QEMU CPU threads were not pinned
3. QEMU was not using huge page backed memory
4. emulator and io threads of QEMU were not pinned.
5. Inside VM the kernel boot parameter was set to 1GB, which was causing TLB to miss on the host.

Corrected configuration:

1. setup host with 4 * 1GB huge page on host
2. edit qemu XML to reflect VCPU, iothread, emulator threads on desired host CPU
3. edit qemu to use host 4 * 1GB page
4. edit VM grub to have isolate CPU and use 2MB pages.
5. run DPDK application using isolated core on VM
6. taskset firecracker thread on ubuntu

We were able to achieve around 3.5X to 4X performance on the current DPDK code.

note: there is a lot of room to improve code performance for DPDK primary and secondary application also.

Intel Xeon E5-2620 is Sandy Bridge CPU which officially supports AVX and not AVX2.

DPDK 20.11 meson build, ninja -C build will generate code with AVX instructions and not AVX2. But (Based on the live debug) PKTGEN forces the compiler to add AVX2 to be inserted, thus causing illegal instruction.

Solution: edit meson.build in line 22

from

Based on the logical cores specified as part of EAL Arguments, one can choose either a single or multiple cores for DPDK LCORE. The DPDK API rte_eal_remote_launch helps to start a specific function with arguments under DPDK context on specific LCORE. If not invoked with the API, the thread will be in wait state (created and waiting to run user application). One should use the API in the context of the master core for launching new functions to be running under DPDK EAL context.

Can I assign it to slave threads?

[Answer] One run alternative threads or worker threads other than DPDK lcore threads also by using

1. Service cores using rte_service API
2. non-dpdk or non-eal threads using rte_thread_register

Please refer the DPDK documentation for sample with explanation for use of rte_remote_launch