Performance Tuning

Cockpit for Centos and RHEL 7: Install and Configure

/ Christopher Paquin / Leave a comment

Snail_On_White_Background_600

Introduction

I have recently purchased 3 Dell servers, and put myself to task to build out a new lab. My old lab was in desperate need of updating as I had long past the time when 48GB of memory per node was sufficient. The cost of memory, old or new was not even closely in line with cheap server grade CPUs that were perfect for lab servers. Today you can buy a used E7540, a low power, 12 core (HT enabled) Xeon for less than $30 (USD) from a reputable retailer. Cram two of these into an 11 gen Dell and you are in business.

So, three new (to me) Dell rackmounts, deployed as virtualization servers, and I want a simple way to view performance stats in a nice clean single pain of glass. I am not in any way shape or form looking to build fancy dashboard and setup any sort of historical monitoring. I just want to know where the performance hot spots are when my environment seems to be running slowly.

I installed Cockpit before on a laptop or two and thought it might foot the bill, especially since you could use one dashboard for multiple nodes.

So here we are going to deploy Cockpit on all three nodes, on each the steps are the same.

Prerequisites

First we must open a firewall port on each node.

Continue reading →

How to Install TestPMD on RHEL 7.x

/ Christopher Paquin / Leave a comment

dpdk (1).png

TestPMD is a lightweight application running in user space, utilizing ovs-dpdk, that can be used for testing DPDK in packet forwarding mode.

In this example we want to setup TestPMD on a RHEL VM running in our SR-IOV capable Red Hat OpenStack 10 overcloud. Our passthrough adapters are Intel X520s. Our plan here is to run performance tests via an external load generator.

Before we can get started we need to build a test VM.

VM Details

  • RHEL 7.x
  • Two VFs
    • eth0 – ssh access via admin network
    • eth1 – load generator private network
  • 4 vCPUs
  • 4096 MB Mem
  • 150 GB Disk

Deploy RHEL VM

Your first step is to deploy your RHEL VM, configure your primary network interface (eth0 for ssh) via VM console. Eth1 needs to be up and configured to start at boot, but do not assign it an IP address. Next, register your VM with your local satellite server or with RH CDN.

Download DKDP

Use the link below to download the “Latest Major” version of DPDK. Place the tarfile in /root on the VM and untar.

Install Prerequisites

Before we can compile DPDK, we need to install a few prereqs.

Install gcc

#yum -y install gcc

Install lubnuma-devel

#yum -y install glibc-devel

Install Kernel Headers and Devel

#yum -y install kernel-headers.x86_64 kernel-devel.x86_64

Install NUMA Packages

#yum -y install numad.x86_64 numactl-libs.x86_64 numactl-devel.x86_64

Install libpcap

#yum -y install libpcap.x86_64 libpcap-devel.x86_64

Install Tuned Profiles

#yum -y install tuned-profiles-cpu-partitioning.noarch

Install DPDK Tools Package

#yum -y install dpdk-tools.x86_64

Compile DPDK

Compile using the “Quick Start” guide below
http://dpdk.org/doc/quick-start

Determine the PCI address of your test interface using ethtool.

# ethtool -i eth1
driver: ixgbevf
version: 1.5.10-k
firmware-version: 5.02 0x80002390 1.1313.0
expansion-rom-version:
bus-info: 0000:00:05.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Here the PCI address for eth1 is 0000:00:05.0

Load DPDK Driver for Testing Interface

In this test we are using the Intel x520 NIC, which is directly accessible to our VM via SR-IOV passthrough. If you are passing through a different NIC, your process will differ.

#modprobe vfio-pci
#dpdk-devbind –bind=vfio-pci 0000:00:05.0

Verify Driver using dpdk-devbind

# dpdk-devbind –s

Network devices using DPDK-compatible driver
============================================
0000:00:05.0 ‘82599 Ethernet Controller Virtual Function’ drv=igb_uio unused=ixgbevf,vfio-pci

Network devices using kernel driver
===================================
0000:00:03.0 ‘Virtio network device’ if=eth0 drv=virtio-pci unused=virtio_pci,igb_uio,vfio-pci *Active*

Example TestPDM Start Script

In the example script below, we are going to start TestPDM. By default, TestPDM will forward any packets recieved on eth1 back to the sending MAC

#!/bin/bash

#VARS
NICADDRESS1=’0000:00:05.0′

/root/dpdk-17.08/build/app/testpmd -l 0,1,2,3 –socket-mem 512 -n 4 –proc-type auto –file-prefix pg -w $NICADDRESS1 — –disable-rss –nb-cores=2 –portmask=1 –rxq=1 –txq=1 –rxd=256 –txd=256 –port-topology=chained –forward-mode=macswap -i –auto-start

The example below works for testing an MTU up to 9200 bytes

#works for 9200 byte packet
/root/dpdk-17.08/x86_64-native-linuxapp-gcc/app/testpmd –log-level 8 –huge-dir=/mnt/huge -l 0,1,2,3 -n 4 –proc-type auto –file-prefix pg -w $NICADDRESS1 — –disable-rss –nb-cores=2 –portmask=1 –rxq=2 –txq=2 –rxd=256 –txd=256 –port-topology=chained –forward-mode=mac –eth-peer=0,00:10:94:00:00:06 –mbuf-size=10240 –total-num-mbufs=32768 –max-pkt-len=9200 -i –auto-start

 

TestPMD has a plethora of options.

For additional information on the options used above, refer to the user guide.

Additional Resources

 

Deploying a Non-NUMA Aligned Instance in RHEL OSP

/ Christopher Paquin / Leave a comment

amd_45nm_2.jpg

When you Nova boot an instance in an OSP Overcloud deployed using SR-IOV and CPU Pinning, that instance will be NUMA aligned, meaning that its vCPUs, memory, and SR-IOV VF (Virtual Function) will all be local to the same NUMA node.  Nova will not allow you to deploy a non-NUMA aligned instance in such an environment (which you might want to do when testing the cross-NUMA penalty of your hardware).

If you are looking to misalign an instance for the purpose of testing, you can use the following process.

First, ssh to the Compute node running your test instance.  In the example below, we can see that the libvirt shows VM three as being pinned to odd numbered cores. On this hardware that means, NUMA node 1.

In this example, our VM has 4 cores.

# virsh vcpupin 3
VCPU: CPU Affinity
——————————
   0: 11
   1: 35
   2: 17
   3: 41
Using “virsh vcpupin“ you can move the pinned VM cores to the pCPUs local to numa node 0.
In the example below, we start my migrating vCPU 0 to pCPU 12.
# virsh vcpupin 3 0 12
Now we migrate vCPU 1 to pCPU 14.
# virsh vcpupin 3 1 14
Now we migrate vCPU 2 to pCPU 18.
# virsh vcpupin 3 2 18
Now we migrate vCPU 3 to pCPU 20.
# virsh vcpupin 3 3 20
Now, pinning now appears as follows.
# virsh vcpupin 3
VCPU: CPU Affinity
———————————-
   0: 12
   1: 14
   2: 18
   3: 20

Roxio Toast Titanium on Mac OS X opening / running very slow – hello DTrace!

/ Christopher Paquin / Leave a comment

ToastIf you find Toast is running very slow on your Mac and you've tried all the other fixes, check to make sure you don't have any mounts defined on your Mac as Toast will continue to try and mount them ad nauseum – thereby causing the slowness.  You can check Directory Utility to see if there are mounts you forgot about.

If that's not it, it is time to embrace the built-in DTrace tools Mac has for your computing pleasure.  This can help you track the calls Roxio makes when it's appears to be running dog slow.

Dtruss is very much like truss or strace – it will show you system calls made by whatever you are dtrussing.  If what you are trying to figure out is shortlived, execsnoop is also your friend as it will show you system calls for any new processes as they are spawned.

Dtruss and execsnoop show when Toast starts up that automountd is always called, thus stale NFS mounts could be causing you headaches.

RedHat Performance Tuning: The Man Pages

/ Christopher Paquin / Leave a comment

OTF365091S This my first post following my week long Redhat Monitoring and Performance Tuning Course (RH442). This course was rather intense and dealt mostly with concepts and theories rather and was not at all a how-to on performance tuning. I am going to do my best to document some of what I learned.  Lets start with the documentation…

Man Pages, are the extensive documentation that comes installed by default on a linux system. They are seperated into 9 books or sections.

  1. General Commands
  2. System Calls
  3. C Library Functions
  4. Special FIles and Drivers
  5. FIle Formats
  6. Games/Screensavers
  7. Protocols
  8. System Admin Commands and Daemons

To access a particular book you specify the book name after the man command. For example:

#man 1 printf

which is different than…

#man 3 printf

To find all the man pages related to printf, you run a the following

#man -k printf

Which gives you the output below listing all man pages related to printf. These specific ones are all on the the printf command specifically. Similarly you can use the apropos command and get the same output

To see only the man pages for printf run the command below

#man -f printf

Which will return only what you see below.

printf (1)           – format and print data
printf (1p)          – write formatted output
printf (3)           – formatted output conversion
printf (3p)          – print formatted output