dpdk | SPDK mirror of DPDK

1. Looks like the program does not invoke rte_eth_dev_configure() before invoking rss_setup(). This is not correct as rte_eth_dev_configure() "must be invoked first before any other function in the Ethernet API" (doc reference). Please re-arrange the program to first invoke rte_eth_dev_configure() with rx_adv_conf.rss_conf and ETH_MQ_RX_RSS specified, then proceed with the rest of bring-up.

2. In rss_setup(), the program invokes rte_eth_dev_rss_reta_update(). For the PMD in question, this API can only be invoked in started state (code reference). So please re-arrange the program a bit more to invoke rte_eth_dev_start() before rss_setup().

3. Please make sure that TCP source port numbers (and/or IP source addresses) are randomised at the sender side. If IP/TCP source and destination fields do not vary, it's no surprise that the packets end up in the same Rx queue.

Adding iommu=pt intel_iommu=on to the grub solve the softlockup and the 10 G Rx

Since v18.05-rc1, there has been item type RTE_FLOW_ITEM_TYPE_ARP_ETH_IPV4. That being said, it might be unsupported by the PMD in question.

Consider matching on the EtherType field instead:

The main reason why DPDK ports are not identified in your environment is because the NIC in use is not having a supported vendor Poll Mode Driver. Please refer to list of supported NIC from various vendor Realtek is not among them.

Work around: you can use PCAP PMD to solve the problem. Please follow the steps for your environment as

1. ensure your ethernet port is linked to kernel driver r8169
2. isntall libpacp-dev for your enviorment (linux/bsd)
3. rebuild dpdk with PCAP support.
4. start your application with following args --no-pci --vdev=net_pcap0,iface=enp1s0

This will use libpcap PMD as wrapper for RX and TX over non supported NIC

As per the question

RSS did not have a good load balance between CPU cores case the rx packets has been modified by insert tags between mac and ip

There are couple of items which needs to be clarified. so let me explain

1. Certain Physical and virtual NIC exposes RSS via DPDK RX offload for fixed tuples like IP, Protocol, TCP|UDP|SCTP port number.
2. Certain NIC allows to configure the hash reta algorithm to better suit to needs (example when the source destination IP address is fixed we can skip and use others).
3. As I recollect from DPDK 18.11, RTE_FLOW is been introduced to support RSS on selected RX queues (example Q1,Q2,Q3 can be RSS for TCP packets while Q4,Q5 can be used for UDP). But again this is based on the either Inner or Outer IP+PORT number
4. For DPDK version 19.11 onwards RTE_FLOW is been enhanced to support RAW Pattern. The intend of this feature is support Special protocol which by default the NIC does not understand like (VXLAN, GENEVE, RTP and other protocols).
5. For NIC like Fortville & Columbiaville (from Intel) allows loading of special firmware via DDP (Dynamic Device Personation) to configure special fabric headers or MPLS like headers (between ethernet and ip) to be parsed, lookup and used as seed for RSS (allowing better distribution).
6. There are NIC which do support L2 layer but these would be limited SMAC, DMAC, VLAN1,VLAn2, MPLS only and not custom header.

Hence depending upon NIC, vendor, RSS support for L2 and firmware the ability calculating RSS on fields between varies in port init or RTE_FLOW specific configuration. For example RSS ETH supported on

1. I40E is I40E_INSET_DMAC | I40E_INSET_SMAC
2. DPAA2 is NH_FLD_ETH_TYPE and NET_PROT_ETH
3. CNXK is RSS_DMAC_INDEX
4. OCTEONX2 is FLOW_KEY_TYPE_VLAN and FLOW_KEY_TYPE_CH_LEN_90B

Hence for NIC ixgbe and XXV710 there is no ready support for the custom header between ethernet and IP.

Alternatives:

1. Use a smart NIC or FPGA: that is programmed to parse and RSS on your specific headers to RSS on multiple RX queue
2. Work with Intel using XXV710 (Fortville): to create DDP which can parse your specific headers as RSS on multiple RX queue.
3. Identify DPDK NIC: which can parse RAW header as defined 12.2.6.2. Once the support is added by the vendor you can create a simple traffic spread tool which will ensure traffic distribution across the desired RX queues in round robin fashion.
4. Use SW to support the missing hardware.

note: I am not recommending use of HW based static Round Robin as it will create 2 fold problem

• if it is pure DPDK BOND Round ROBIN you will not have any flow pinning
• If you use hash based pinning, there are chances elephant flow can pushed 1 or a few queues causing performance drops in cpu processing.
• My recommendation is to use EVENTDEV model with atomic mode, which ensure better cache locality (at a given instance same flows will fall onto worker thread) and almost linear performance performance. sample app

For option 4 (Software Model):

1. Disable RSS in port_init function
2. Use single RX queue to receive all the packet either custom rx thread.
3. calculate the hash based on the desired header and update mbuf hash field.
4. Use rte_distributor library to spread traffic based on the custom.
5. or use rte_eventdev with atomic model to spread the work load on multiple worker.

[Clarification from Comments]:

1. I have asked relevant practitioners， they said modifying the pmd driver can solves my problem, its the only way?

[ANSWER] Since you are using custom header and not generic VLAN|IP|Port this suggestion is not correct. As you have clarified in question and comments you want to distribute like RSS for custom header.

1. I haven't written any code about rte_flow distribution yet, i read the rte_flow example, don't see the code to configure round-robin

[ANSWER] as explained above not all nic support RSS and RAW. Since your current NIC is ixgbe and i40e the function of parsing and executing RSS for custom header is unlikely. you can try option 2 (work with intel create new ddp) for i40e to achieve the same or implement in SW as suggested in option 4.

1. im not ask for solution, I just want to know how to set up round robin by 'RTE_FLOW' Can you give me a few API

[ANSWER] normally one updates with code snippets or steps used for reproducing the error. But the current question is more like clarification. Please refer above.

[EDIT-1] clarification of using same interface for DPDK and Kernel routing

Answer> as discussed over comments please refer to DPDKD + kernel on same interface

Based on the information shared there are multiple questions to the single query I'm trying to bypass all traffic to the PMD I'm currently developing. Addressing each one separately below

1. question 1: using dpdk-l2fwd example application

Answer> DPDK application l2fwd application makes use of basic APi with almost no HW offloads. Based on your environment (I have a board with one ethernet interface (eth0)), the right set of parameters should be -p 0x1 --no-mac-updating -T 1. This will configure the application to receive and transmit packet using single DPDK interface (that is eth0 on your board).

Note: DPDK Application can work with DPDK PMD both physical and virtual

1. question 2: I tried to use net_tap, using the following command:

Answer> If the intend is to intercept the traffic from physical and then forward to tap interface, then one needs modify the eal arguments as ./build/l2fwd --vdev=net_tap0,iface="my_eth0" -- -p 0x3 -T 1 --no-mac-updating. This will allow the application to probe physical NXP interface (eth0) and make use of Linux TAP interface as secondary interface. Thus any traffic from NXP and TAP will be cross connected such as NXP (eth0) <==> TAP (my_eth0)

1. question 3: ./usertools/dpdk-devbind.py --status returns empty

Answer> Form the dpdk site supported NIC list NXP dpaa, dpaa2, enetc, enetfec, pfe. Cross checking the kernel driver fsl_dpaa2_eth I think it is safe to assume dpaa2 PMD is supported. As you have mentioned the NIC is not enumerated, it looks like there are certain caveats to such model revision, supported board, BSP package, vendor-sub vendor ID check etc. More details can be found Board Support Package, and DPAA2 NIC guide

Debug & Alternative solutions:

1. To start with use the Kernel Driver to bring in packets
2. Use extra logging and debug to identify why the NIC is shown in the application

Approach 1:

1. Make sure the NIC is bind with kernel driver fsl_dpaa2_eth.
2. ensure NIC is connected and link is up with ethtool eth0
3. set to promiscous mode with ifconfig eth0 promisc up
4. start DPDK application with PCAP PMD, ./build/l2fwd --vdev=net_pcap0,iface=eth0 -- -p 1 --no-mac-updating -T 1
5. Check packet are received and redirected to PCAP eth0 PMD by checking the statistics.

Approach 2:

1. Ideally the NIC should be categorized under network device to be probed by debind.py.
2. Check the device details using lshw -c net -businfo for network.
3. try checking with lspci -Dvmmnnk [PCIe BUS:Slot:Function id] for network details.
4. If above details does not show up as network device this might be reason for not getting listed.

• Suggestions or workaround: You can try to forcefully bind with igb_uio or vfio-pci (I am not much famialr with NXP SoC) by dpdk-devbind -b vfio-pci [PCIe S:B:F]. Then cross check with lspci -ks [PCIe S:B:F]. Once successfully done, one can start dpdk l2fwd in PMD debug mode with ./build/l2fwd -a [PCIe S:B:F] --log-level=pmd,8 -- -p 1 --no-mac-updating | more. Thus by intercepting and interpreting the logs one can identify what is going

Note:

1. It is assumed the application is build with static libraries and not dynamic. To build with static libraries use make static for l2fwd.
2. For the described use case recommended application is basicfwd/skeleton rather than l2fwd.

Support for GRE and/or Inner RSS is touch and go for many NIC, so most NIC does not work out of box. For example

1. For Intel Fortville (10gbps, 25Gbps, 40Gbps) NIC: one needs to update firmware and DDP profile to parse and compute inner RSS.
2. For Intel Columbiaville (100Gbps, 50Gbps, 25Gbps) NIC: one needs to update firmware and then update driver. As I recollect default DDP profile parses GRE.
3. In case of Mellanox MLX5 (there are multiple variant for ConnectX-5 and Connect-6), some of them support GRE based parsing and RSS, while other require ESWITCH to perform such actions.

Hence using testpmd one needs to

1. enable MQ_RSS in port and RX queue configuration
2. For specific (inner RSS) enable via RTE_FLOW API

With MT2892 Family [ConnectX-6 Dx] one can enable Inner 5 tuple RSS for GRE encapsulated packet with testpmd.

1. Start packet generator (DPDK pktgen) use sudo ./usr/local/bin/pktgen --file-prefix=3 -a81:00.0,txq_inline_mpw=128 -l 6-27 -- -P -m "[8-11:12-27].0" -N -s"0:rtp_balanced_gre.pcap"
2. Start testpmd in interactive mode with multiple RX queues using dpdk-testpmd --socket-mem=1024 --file-prefix=2 -l 7,8-23 -a0000:41:00.1,mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1,rxq_cqe_comp_en=4 -- --port-numa-config=0,0 --socket-num=0 --burst=128 --txd=8192 --rxd=8192 --mbcache=512 --rxq=16 --txq=16 --nb-cores=8 -a --forward-mode=io --numa --rss-udp --enable-rx-cksum --no-mlockall --no-lsc-interrupt -a --enable-rx-cksum --enable-drop-en --eth-link-speed=100000 --no-rmv-interrupt -i
3. In interactive shell configure the rule as flow create 0 ingress pattern eth / ipv4 / gre / eth / ipv4 / udp / end actions rss level 2 types ip udp end queues 0 1 2 3 end / end.

Note:

1. Ensure to use the right format in match field eth / ipv4 / gre / eth / ipv4 / udp /
2. if the rule is not set on the device only device RX queue0 will be receiving the packets.

Based on the intercept scenario, there are 4 options to solve this problem

1. Option-1: for Remote Termination use of single DPDK port with MAC|VLAN address modification.
2. Option-2: for Remote Termination use of two DPDK port with no MAC address modification.
3. Option-3: for Local Termination use of DPDK TAP PMD to terminate to HOST/GUEST OS, with no packet MAC|VLAN modification.
4. Option-4: for Local Termination use of DPDK PCAP PMD to intercept the packets from Kernel interface, with no packet MAC|VLAN modification.

Please follow the steps to capture the traffic appropriately. For

Option-1:

1. Create the VM with 1 DPDK port.
2. Use testpmd or sample application l2fwd (modified with DPDK rte_pdump_init) to capture and replay back into PORT.
3. Since MAC-address|VLAN is modified with appropriate rules one can forward from remote HOST to desired destination
4. Start dpdk PDUMP application, example sudo ./build/app/dpdk-pdump -- --pdump 'port=0,queue=*,tx-dev=./tx.pcap' . This will capture packets that enter from outside and write into tx.pcap.

Option-2:

1. Create VM with 2 DPDK ports
2. Start DPDK modified application (use dpdk rte_pdump_init) either skeleton or l2fwd.
3. for l2fwd use option --no-mac-updating.
4. Start dpdk PDUMP application, example sudo ./build/app/dpdk-pdump -- --pdump 'port=0,queue=*,tx-dev=./tx.pcap' . This will capture packets that enter from outside and write into tx.pcap

Option-3:

1. Create the VM with 1 DPDK port (AMAZON enic)
2. bind the desired DPDK port (WAN) with appropriate UIO driver
3. Start DPDK application like basicfwd sudo ./build/basicfwd -l 2 --vdev=net_tap0,mac="mac address of DPDK port" -a .
4. Within linux terminal sudo ifconfig dtap0 up
5. [optional] add desired route entry.
6. use linux tcpdump, python or any packet capture application to grab packets from dtap0 interface for RX|TX or both direction.

Option-4:

1. Create the VM with 1 DPDK port (AMAZON enic)
2. make sure the interface is bind with kernel driver.
3. ensure the interface is able to reach outside network (WAN) using ping -I stackoverflow.com
4. run dpdk modified application like basicfwd or l2fwd with rte_pdump_init (example ./build/l2fwd -l 2 --vdev=net_pcap0,iface= -- -p 1 -T 1 --no-mac-updating)
5. capture the packets using DPDK PDUMP applciation. example: sudo ./build/app/dpdk-pdump -- --pdump 'port=0,queue=*,tx-dev=./tx.pcap'

Note:

• refer for use DPDK PDUMP use case.
• recommendation is use option 3, as it eliminates the need for DPDK secondary process and disabling ASLR.

L3fwd-acl works with IP packets only, for all non IP packets it drops. Hence with TAP PMD I had to follow the changes as

1. Disable any NIC PCIe device using --no-pci
2. Use new commands as sudo ./build/l3fwd-acl --file-prefix=mytest -l 2-4 --vdev=net_tap0 --vdev=net_tap1 --no-pci -- -P -p 3 --config="(0,0,4),(1,0,3)" --rule_ipv4=ipv4-sample.db --rule-ipv6=ipv6-sample.db --eth-dest=0, --eth-dest=1,
3. edit ip-sample.db file with the entries to be assigned for dtap0 and dtap1.
4. refer section 24.1.3 to identify the format and for default drop rule
5. start the application and use the following netns commands.