RHEL OSP 10/11 – OVS+DPDK Tunables

93905-6a00e551c39e1c883401a511e32c88970c-pi

Tunables for Dell R630s for use when deploying OVS+DPDK


# OSP 10/11 DPDK Tunables
#
# R630 NUMA locality - CPUs
# node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22
# 24 26 28 30 32 34 36 38 40 42 44 46
#
# node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23
# 25 27 29 31 33 35 37 39 41 43 45 47
#
#
# R630 NUMA locality - NIC
# node 0 dpdk interface - p3p1
# node 1 dpdk interface - p1p1
#
#
#
# NovaVcpuPinSet (OSP 10+)
# These are the cores that Nova will use for scheduling instances. Pair sibling threads together.
# Using cores from NUMA node 0 only to prevent crossing NUMA boundaries
NovaVcpuPinSet: "'4,6,8,10,12,14,16,18,20,22,28,30,32,34,36,38,40,42,44,46'"
#
# NeutronDpdkCoreList (OSP 10/11) OvsPmdCoreList (OSP 12+)
# This parameter configures a list of CPU cores to be used by the OVS-DPDK Poll Mode Drivers
# The first core from a CPU, should be reserved for host processes, and should be excluded from this list.
NeutronDpdkCoreList: "'2,26,3,27'"
#
# HostIsolatedCoreList (OSP 10/11) IsolCpusList (OSP 12+)
# A set list or range of cores (and their sibling threads) to be appended to the tuned cpu-partitioning profile and isolated from the host.
# These cores will be isolated from any host processes
# Assuming you want to isolate nova cores from all system processes, NovaVcpuPinSet + NeutronDpdkCoreList = HostIsolatedCoreList
HostIsolatedCoreList: "'2,3,4,6,8,10,12,14,16,18,20,22,26,27,28,30,32,34,36,38,40,42,44,46'"
#
# HostCpusList (OSP 10/11) & OvsDpdkCoreList (OSP 12+)
# A list of logical cores used by OVS-DPDK processes for dpdk-lcore-mask for non-datapath operations
# These cores must be mutually exclusive from the list of cores in NeutronDpdkCoreList/OvsPmdCoreList and NovaVcpuPinSet.
# Allocate the first physical core (and sibling thread) from each NUMA node irrespective of DPDK interface NUMA locality.
HostCpusList: "'0,24,1,25'"
#
# Provide the number of memory channels in the format - [allowed_pattern: "[0-9]+"]:
NeutronDpdkMemoryChannels: "4"
#
# Set the memory allocated for each socket:
NeutronDpdkSocketMemory: "'2048,2048'"
#
# An array of filters used by Nova to filter a node.These filters will be applied in the order they are listed,
# so place your most restrictive filters first to make the filtering process more efficient.
NovaSchedulerDefaultFilters: "RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter,NUMATopologyFilter"
#
# Kernel arguments for Compute node
ComputeKernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on"

 

How to disable Cloud-Init in a RHEL Cloud Image

happy_cloud1600

So this one is pretty simple. However, I found a lot of misinformation along the way, so I figured that I would jot the proper (and most simple) process here.

Symptoms: a RHEL (or variant) VM that takes a very long time to boot. On the VM console, you can see the following output while the VM boot process is stalled and waiting for a timeout. Note that the message below has nothing to do with cloud init, but its the output that I have most often seen on the console while waiting for a VM to boot.

[106.325574} random: crng init done

Note that I have run into this issue in both OpenStack (when booting from external provider networks) and in KVM.

Upon initial boot of the VM, run the command below.

touch /etc/cloud/cloud-init.disabled

Seriously, that’s it. No need to disable or remove cloud-init services. See reference.

 

 

How to Install TestPMD on RHEL 7.x

dpdk (1).png

TestPMD is a lightweight application running in user space, utilizing ovs-dpdk, that can be used for testing DPDK in packet forwarding mode.

In this example we want to setup TestPMD on a RHEL VM running in our SR-IOV capable Red Hat OpenStack 10 overcloud. Our passthrough adapters are Intel X520s. Our plan here is to run performance tests via an external load generator.

Before we can get started we need to build a test VM.

VM Details

  • RHEL 7.x
  • Two VFs
    • eth0 – ssh access via admin network
    • eth1 – load generator private network
  • 4 vCPUs
  • 4096 MB Mem
  • 150 GB Disk

Deploy RHEL VM

Your first step is to deploy your RHEL VM, configure your primary network interface (eth0 for ssh) via VM console. Eth1 needs to be up and configured to start at boot, but do not assign it an IP address. Next, register your VM with your local satellite server or with RH CDN.

Download DKDP

Use the link below to download the “Latest Major” version of DPDK. Place the tarfile in /root on the VM and untar.

Install Prerequisites

Before we can compile DPDK, we need to install a few prereqs.

Install gcc

#yum -y install gcc

Install lubnuma-devel

#yum -y install glibc-devel

Install Kernel Headers and Devel

#yum -y install kernel-headers.x86_64 kernel-devel.x86_64

Install NUMA Packages

#yum -y install numad.x86_64 numactl-libs.x86_64 numactl-devel.x86_64

Install libpcap

#yum -y install libpcap.x86_64 libpcap-devel.x86_64

Install Tuned Profiles

#yum -y install tuned-profiles-cpu-partitioning.noarch

Install DPDK Tools Package

#yum -y install dpdk-tools.x86_64

Compile DPDK

Compile using the “Quick Start” guide below
http://dpdk.org/doc/quick-start

Determine the PCI address of your test interface using ethtool.

# ethtool -i eth1
driver: ixgbevf
version: 1.5.10-k
firmware-version: 5.02 0x80002390 1.1313.0
expansion-rom-version:
bus-info: 0000:00:05.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Here the PCI address for eth1 is 0000:00:05.0

Load DPDK Driver for Testing Interface

In this test we are using the Intel x520 NIC, which is directly accessible to our VM via SR-IOV passthrough. If you are passing through a different NIC, your process will differ.

#modprobe vfio-pci
#dpdk-devbind –bind=vfio-pci 0000:00:05.0

Verify Driver using dpdk-devbind

# dpdk-devbind –s

Network devices using DPDK-compatible driver
============================================
0000:00:05.0 ‘82599 Ethernet Controller Virtual Function’ drv=igb_uio unused=ixgbevf,vfio-pci

Network devices using kernel driver
===================================
0000:00:03.0 ‘Virtio network device’ if=eth0 drv=virtio-pci unused=virtio_pci,igb_uio,vfio-pci *Active*

Example TestPDM Start Script

In the example script below, we are going to start TestPDM. By default, TestPDM will forward any packets recieved on eth1 back to the sending MAC

#!/bin/bash

#VARS
NICADDRESS1=’0000:00:05.0′

/root/dpdk-17.08/build/app/testpmd -l 0,1,2,3 –socket-mem 512 -n 4 –proc-type auto –file-prefix pg -w $NICADDRESS1 — –disable-rss –nb-cores=2 –portmask=1 –rxq=1 –txq=1 –rxd=256 –txd=256 –port-topology=chained –forward-mode=macswap -i –auto-start

The example below works for testing an MTU up to 9200 bytes

#works for 9200 byte packet
/root/dpdk-17.08/x86_64-native-linuxapp-gcc/app/testpmd –log-level 8 –huge-dir=/mnt/huge -l 0,1,2,3 -n 4 –proc-type auto –file-prefix pg -w $NICADDRESS1 — –disable-rss –nb-cores=2 –portmask=1 –rxq=2 –txq=2 –rxd=256 –txd=256 –port-topology=chained –forward-mode=mac –eth-peer=0,00:10:94:00:00:06 –mbuf-size=10240 –total-num-mbufs=32768 –max-pkt-len=9200 -i –auto-start

 

TestPMD has a plethora of options.

For additional information on the options used above, refer to the user guide.

Additional Resources

 

Deploying a Non-NUMA Aligned Instance in RHEL OSP

amd_45nm_2.jpg

When you Nova boot an instance in an OSP Overcloud deployed using SR-IOV and CPU Pinning, that instance will be NUMA aligned, meaning that its vCPUs, memory, and SR-IOV VF (Virtual Function) will all be local to the same NUMA node.  Nova will not allow you to deploy a non-NUMA aligned instance in such an environment (which you might want to do when testing the cross-NUMA penalty of your hardware).

If you are looking to misalign an instance for the purpose of testing, you can use the following process.

First, ssh to the Compute node running your test instance.  In the example below, we can see that the libvirt shows VM three as being pinned to odd numbered cores. On this hardware that means, NUMA node 1.

In this example, our VM has 4 cores.

# virsh vcpupin 3
VCPU: CPU Affinity
——————————
   0: 11
   1: 35
   2: 17
   3: 41
Using virsh vcpupin you can move the pinned VM cores to the pCPUs local to numa node 0.
In the example below, we start my migrating vCPU 0 to pCPU 12.
# virsh vcpupin 3 0 12
Now we migrate vCPU 1 to pCPU 14.
# virsh vcpupin 3 1 14
Now we migrate vCPU 2 to pCPU 18.
# virsh vcpupin 3 2 18
Now we migrate vCPU 3 to pCPU 20.
# virsh vcpupin 3 3 20
Now, pinning now appears as follows.
# virsh vcpupin 3
VCPU: CPU Affinity
———————————-
   0: 12
   1: 14
   2: 18
   3: 20

OpenStack: Rabbitmq Cannot Join Cluster, Already a Member

rabbitmq-sh-600x600

You can run into this error when attempting to join a node into a Rabbitmq cluster when the cluster believes that a particular node is already a member of the cluster. I have run into this issue a few times and is usually seen when attempting to recover from a crash of an OpenStack controller.

I have run into this issue a few times and is usually seen when attempting to recover from a crash of an OpenStack controller.

Below are the steps to resolve the issue.

The error below is seen when attempting to add a node back into the cluster.

INFO REPORT==== 27-Jan-2017::16:57:22 ===
Already member of cluster: [rabbit@nodectrl2,rabbit@nodectrl1,
rabbit@nodectrl0]

We check the cluster status for confirmation.

root@nodectrl1 rabbitmq]# rabbitmqctl cluster_status
Cluster status of node rabbit@nodectrl1 …
[{nodes,[{disc,[rabbit@nodectrl0,rabbit@nodectrl1,
rabbit@nodectrl2]}]},
{running_nodes,[rabbit@nodectrl2,rabbit@nodectrl1]},
{cluster_name,<<“rabbit@nodectrl0.localdomain”>>},
{partitions,[]},
{alarms,[{rabbit@nodectrl2,[]},{rabbit@nodectrl1,[]}]}]

Now we force the cluster to forget the affected node.

[root@nodectrl1 rabbitmq]# rabbitmqctl forget_cluster_node rabbit@nodectrl0
Removing node rabbit@nodectrl0 from cluster …

We then check the cluster status to ensure that it has been removed from the cluster.

[root@nodectrl1 rabbitmq]# rabbitmqctl cluster_status

Cluster status of node rabbit@nodectrl1 …
[{nodes,[{disc,[rabbit@nodectrl1,rabbit@nodectrl2]}]},
{running_nodes,[rabbit@nodectrl2,rabbit@nodectrl1]},
{cluster_name,<<“rabbit@nodectrl0.localdomain”>>},
{partitions,[]},
{alarms,[{rabbit@nodectrl2,[]},{rabbit@nodectrl1,[]}]}]

We can now add our node back into the cluster.

[root@nodectrl1 rabbitmq]#  rabbitmqctl -n nodectrl1 join_cluster rabbit@nodectrl0.localdomain

OpenStack: instackenv.json Format Example

stack-example

Here is a quick and dirty example of the format of your instackenv.json. This is the file that Ironic uses to import nodes.

Enter your IPMI user id under “pm_user”

Enter your IPMI password under “pm_password”

 

{
"nodes":[
    {
        "mac":[
            "74:E6:E2:FB:71:B0"
        ],
        "cpu":"4",
        "memory":"6144",
        "disk":"40",
        "arch":"x86_64",
        "name":"control01",
        "pm_type":"pxe_ipmitool",
        "pm_user":"admin",
        "pm_password":"admin",
        "pm_addr":"10.75.99.120"
    },
    {
        "mac":[
            "74:E6:E2:FB:71:D6"
        ],
        "cpu":"4",
        "memory":"6144",
        "disk":"40",
        "arch":"x86_64",
        "name":"control02",
        "pm_type":"pxe_ipmitool",
        "pm_user":"admin",
        "pm_password":"admin",
        "pm_addr":"10.75.99.119"
    },
    {
        "mac":[
            "74:E6:E2:FB:73:D0"
        ],
        "cpu":"4",
        "memory":"6144",
        "disk":"40",
        "arch":"x86_64",
        "name":"control03",
        "pm_type":"pxe_ipmitool",
        "pm_user":"admin",
        "pm_password":"admin",
        "pm_addr":"10.75.99.118"
    },
    {
        "mac":[
            "74:E6:E2:FB:27:D4"
        ],
        "cpu":"4",
        "memory":"6144",
        "disk":"40",
        "arch":"x86_64",
        "name":"compute01",
        "pm_type":"pxe_ipmitool",
        "pm_user":"admin",
        "pm_password":"admin",
        "pm_addr":"10.75.99.117"
    },
]
}

OpenStack: How to Rebuild Ironic’s Database

cropped-97743-computer-room

Move the current database to a backup location for safe keeping.

# mv /var/lib/ironic-inspector/inspector.sqlite /root/

Restart Ironic services. Here we are hitting them all to be safe.

# for service in `systemctl list-unit-files | grep ironic | awk ‘{print $1}’`; do  systemctl restart $service; done

Now rebuild the database.

# ironic-inspector-dbsync –config-file /etc/ironic-inspector/inspector.conf upgrade