How to Install TestPMD on RHEL 7.x

dpdk (1).png

TestPMD is a lightweight application running in user space, utilizing ovs-dpdk, that can be used for testing DPDK in packet forwarding mode.

In this example we want to setup TestPMD on a RHEL VM running in our SR-IOV capable Red Hat OpenStack 10 overcloud. Our passthrough adapters are Intel X520s. Our plan here is to run performance tests via an external load generator.

Before we can get started we need to build a test VM.

VM Details

  • RHEL 7.x
  • Two VFs
    • eth0 – ssh access via admin network
    • eth1 – load generator private network
  • 4 vCPUs
  • 4096 MB Mem
  • 150 GB Disk

Deploy RHEL VM

Your first step is to deploy your RHEL VM, configure your primary network interface (eth0 for ssh) via VM console. Eth1 needs to be up and configured to start at boot, but do not assign it an IP address. Next, register your VM with your local satellite server or with RH CDN.

Download DKDP

Use the link below to download the “Latest Major” version of DPDK. Place the tarfile in /root on the VM and untar.

Install Prerequisites

Before we can compile DPDK, we need to install a few prereqs.

Install gcc

#yum -y install gcc

Install lubnuma-devel

#yum -y install glibc-devel

Install Kernel Headers and Devel

#yum -y install kernel-headers.x86_64 kernel-devel.x86_64

Install NUMA Packages

#yum -y install numad.x86_64 numactl-libs.x86_64 numactl-devel.x86_64

Install libpcap

#yum -y install libpcap.x86_64 libpcap-devel.x86_64

Install Tuned Profiles

#yum -y install tuned-profiles-cpu-partitioning.noarch

Install DPDK Tools Package

#yum -y install dpdk-tools.x86_64

Compile DPDK

Compile using the “Quick Start” guide below

Determine the PCI address of your test interface using ethtool.

# ethtool -i eth1
driver: ixgbevf
version: 1.5.10-k
firmware-version: 5.02 0x80002390 1.1313.0
bus-info: 0000:00:05.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Here the PCI address for eth1 is 0000:00:05.0

Load DPDK Driver for Testing Interface

In this test we are using the Intel x520 NIC, which is directly accessible to our VM via SR-IOV passthrough. If you are passing through a different NIC, your process will differ.

#modprobe vfio-pci
#dpdk-devbind –bind=vfio-pci 0000:00:05.0

Verify Driver using dpdk-devbind

# dpdk-devbind –s

Network devices using DPDK-compatible driver
0000:00:05.0 ‘82599 Ethernet Controller Virtual Function’ drv=igb_uio unused=ixgbevf,vfio-pci

Network devices using kernel driver
0000:00:03.0 ‘Virtio network device’ if=eth0 drv=virtio-pci unused=virtio_pci,igb_uio,vfio-pci *Active*

Example TestPDM Start Script

In the example script below, we are going to start TestPDM. By default, TestPDM will forward any packets recieved on eth1 back to the sending MAC



/root/dpdk-17.08/build/app/testpmd -l 0,1,2,3 –socket-mem 512 -n 4 –proc-type auto –file-prefix pg -w $NICADDRESS1 — –disable-rss –nb-cores=2 –portmask=1 –rxq=1 –txq=1 –rxd=256 –txd=256 –port-topology=chained –forward-mode=macswap -i –auto-start

The example below works for testing an MTU up to 9200 bytes

#works for 9200 byte packet
/root/dpdk-17.08/x86_64-native-linuxapp-gcc/app/testpmd –log-level 8 –huge-dir=/mnt/huge -l 0,1,2,3 -n 4 –proc-type auto –file-prefix pg -w $NICADDRESS1 — –disable-rss –nb-cores=2 –portmask=1 –rxq=2 –txq=2 –rxd=256 –txd=256 –port-topology=chained –forward-mode=mac –eth-peer=0,00:10:94:00:00:06 –mbuf-size=10240 –total-num-mbufs=32768 –max-pkt-len=9200 -i –auto-start


TestPMD has a plethora of options.

For additional information on the options used above, refer to the user guide.

Additional Resources


Linux: Using Tcpdump to Capture LLDP Info


According to Wikipedia, “Link Layer Discovery Protocol (LLDP) is a vendor-neutral link layer protocol in the Internet Protocol Suite used by network devices for advertising their identity, capabilities, and neighbors on an IEEE 802 local area network, principally wired Ethernet”

LLDP is often what you will find running on non-Cisco switches and routers (which usually run CDP). If you want to use tcpdump to capture northbound switch port information, you can use the example below as a guide.


# tcpdump -nn -v -i p1p2 ether proto 0x88cc
tcpdump: WARNING: p1p2: no IPv4 address assigned
tcpdump: listening on p1p2, link-type EN10MB (Ethernet), capture size 65535 bytes
19:00:12.559556 LLDP, length 218
Chassis ID TLV (1), length 7
Subtype MAC address (4): f4:8e:38:28:b6:87
Port ID TLV (2), length 11
Subtype Interface Name (5): ethernet11
Time to Live TLV (3), length 2: TTL 120s
Port Description TLV (4), length 39: Big Cloud Fabric Switch Port ethernet11
System Name TLV (5), length 22: SW01




Deploying a Non-NUMA Aligned Instance in RHEL OSP


When you Nova boot an instance in an OSP Overcloud deployed using SR-IOV and CPU Pinning, that instance will be NUMA aligned, meaning that its vCPUs, memory, and SR-IOV VF (Virtual Function) will all be local to the same NUMA node.  Nova will not allow you to deploy a non-NUMA aligned instance in such an environment (which you might want to do when testing the cross-NUMA penalty of your hardware).

If you are looking to misalign an instance for the purpose of testing, you can use the following process.

First, ssh to the Compute node running your test instance.  In the example below, we can see that the libvirt shows VM three as being pinned to odd numbered cores. On this hardware that means, NUMA node 1.

In this example, our VM has 4 cores.

# virsh vcpupin 3
VCPU: CPU Affinity
   0: 11
   1: 35
   2: 17
   3: 41
Using virsh vcpupin you can move the pinned VM cores to the pCPUs local to numa node 0.
In the example below, we start my migrating vCPU 0 to pCPU 12.
# virsh vcpupin 3 0 12
Now we migrate vCPU 1 to pCPU 14.
# virsh vcpupin 3 1 14
Now we migrate vCPU 2 to pCPU 18.
# virsh vcpupin 3 2 18
Now we migrate vCPU 3 to pCPU 20.
# virsh vcpupin 3 3 20
Now, pinning now appears as follows.
# virsh vcpupin 3
VCPU: CPU Affinity
   0: 12
   1: 14
   2: 18
   3: 20

Driving in the Fast Lane: Huge Page support in OpenStack Compute

arger page sizes there is also an increased potential for memory to be wasted as processes must allocate memory in pages but not all of the memory on the page may actually be required.

Source: Driving in the Fast Lane: Huge Page support in OpenStack Compute

Asus RT-AC66U: How to Configure DHCP Static IP Reservations for IOT/Smart Home Devices


I’ve never been a fan of using DHCP reservations to reserve an IP address for a device. However, there are a few situations where a static reservation is the best route to take.

Try your hand with Home Automation via devices like the Wink Hub,  IP cameras,  smart plugs,  satellite receivers, or even the Raspberry Pi and you will know full well the right time to use a static IP reservation.

In this post, I am going to walk you through that process on the Asus RT-AC66u.

First, log into your Routers webUI, select “LAN” in the left-hand menu. Then select the “DHCP Server” tab. Scroll midway down the page. Locate “Enable Manual Assignment” and select the “Yes” radio button.



Just below the section above, you will see a drop down box that you can use to select your device using the MAC or currently assigned IP. Enter the “Hostname” for the device. Select “Add/Delete“. This will add your device to the list.



Scroll down and find your device in the list. Select the icon to the left.  Change the “Name” of the device if you so desire. The only really important information here is the “IP” and the “MAC”. 




Here you also can change the default icon for the device, or add your own custom icons. Here, I am using a camera icon that I downloaded.



Below you can see some of the custom icons and names that I have set to help me keep track of my devices.