When you Nova boot an instance in an OSP Overcloud deployed using SR-IOV and CPU Pinning, that instance will be NUMA aligned, meaning that its vCPUs, memory, and SR-IOV VF (Virtual Function) will all be local to the same NUMA node. Nova will not allow you to deploy a non-NUMA aligned instance in such an environment (which you might want to do when testing the cross-NUMA penalty of your hardware).
If you are looking to misalign an instance for the purpose of testing, you can use the following process.
First, ssh to the Compute node running your test instance. In the example below, we can see that the libvirt shows VM three as being pinned to odd numbered cores. On this hardware that means, NUMA node 1.
In this example, our VM has 4 cores.
# virsh vcpupin 3
VCPU: CPU Affinity
——————————
0: 11
1: 35
2: 17
3: 41
Using “virsh vcpupin“ you can move the pinned VM cores to the pCPUs local to numa node 0.
In the example below, we start my migrating vCPU 0 to pCPU 12.
This article documents the configuration used to configure SR-IOV in OSP 8/Liberty on Dell hardware
Compute Node Configuration
This section will outline the changes needed to configure SR-IOV on each Compute Node.
Bios Configuration on Dell Compute Nodes
First, you will need to ssh to the drac of each Compute Node. Then, type the command below to enter racadm command line.
#racadm
Type the command below to enable SRIOV.
#racadm set BIOS.IntegratedDevices.SriovGlobalEnable Enabled
[Key=BIOS.Setup.1-1#IntegratedDevices]
RAC1017: Successfully modified the object value and the change is in
pending state.
To apply modified value, create a configuration job and reboot
the system. To create the commit and reboot jobs, use “jobqueue”
command. For more information about the “jobqueue” command, see RACADM
help.
Type the command below to verify your configuration.
This tells nova that all VFs belonging to the physical interface, “p1p1“, are allowed to be passed through to VMs and belong to the neutron provider network “sriov_net1” and all VFs belonging to the physical interface, “p3p1“, are allowed to be passed through for the network “sriov_net2“.
Restart nova-compute on each compute node with the command shown below.
#systemctl restart openstack-nova-compute
Install and Enable Neutron Sriov-Agent (Compute)
Note that the sriov-agent is not required, however, we are going to install and configure it anyway.
Neutron-Server changes in /etc/neutron/plugins/ml2/ml2_conf_sriov.ini(Controller)
The change below needs to be made in /etc/neutron/plugins/ml2/ml2_conf_sriov.ini on each controller
Update the /etc/neutron/plugins/ml2/ml2_conf_sriov.ini on each controller.
In our case,the vendor_id is 8086 and the product_id is 10ed.
supported_pci_vendor_devs = 8086:10ed
Modify Nuetron-Server Startup
Edit /usr/lib/systemd/system/neutron-server.service. Here we add –config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini to the ExecStart line. See example below.
# cat neutron-server.service
[Unit]
Description=OpenStack Neutron Server
After=syslog.target network.target
Below is a “Post Install Validation Checklist” that I am working on for validating an OpenStack Deployment. This is a work in progress and is sure to be missing some items. I plan to attempt to treat this post as a live document an update as I find new items to add to the checklist
Component
Task
All
On each node (compute, controller) apply post-install configurations. Ensure ssh keys for root and nova exist on all hosts
All
If deploying with Staypuft installer, ensure that puppet is stopped and disabled so that it does not overwrite post-install configurations
Glance
Ensure that you can upload and configure OS Images
Nova
Ensure that you can create flavors
Nova
Ensure that you can successfully provision test instances. Test with each flavor and OS Image
Cinder
Provision and assign a block device from Cinder. Also ensure that you can detach and delete a block device
Neutron
Create and test tenant networks
Neutron
Provision and assign a floating IP from an external network for each tenant network. Ensure that connectivity is successful
Nova
Define and assign a security group to an instance, enabling at least SSH and ICMP access
All
Reboot of all nodes to ensure configuration persistence, document any configuration changes. Push any fixes out to nodes when necessary
Nova
Generate and test keypairs to be used for access to instances
Nova
Ensure you are able to create and delete instance snapshots
Swift
If using swift, verify that you can create object stor and upload or create files
Nova
Test Live Migration by migrating instances across each compute node