Introduction
This article documents the configuration used to configure SR-IOV in OSP 8/Liberty on Dell hardware
Compute Node Configuration
This section will outline the changes needed to configure SR-IOV on each Compute Node.
Bios Configuration on Dell Compute Nodes
First, you will need to ssh to the drac of each Compute Node. Then, type the command below to enter racadm command line.
#racadm
Type the command below to enable SRIOV.
#racadm set BIOS.IntegratedDevices.SriovGlobalEnable Enabled
[Key=BIOS.Setup.1-1#IntegratedDevices]
RAC1017: Successfully modified the object value and the change is in
pending state.
To apply modified value, create a configuration job and reboot
the system. To create the commit and reboot jobs, use “jobqueue”
command. For more information about the “jobqueue” command, see RACADM
help.
Type the command below to verify your configuration.
#racadm>>get BIOS.IntegratedDevices.SriovGlobalEnable
racadm get BIOS.IntegratedDevices.SriovGlobalEnable
[Key=BIOS.Setup.1-1#IntegratedDevices]
SriovGlobalEnable=Enabled
If the server already has an OS, reboot it to make these settings stick. If it doesn’t, use these racadm commands to power cycle.
#racadm serveraction powerdown
#racadm serveraction powerup
Grub Configuration on Compute Nodes
Add “intel_iommu=on” to the GRUB_CMDLINE_LINUX line as shown below.
# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=”$(sed ‘s, release .*$,,g’ /etc/system-release)”
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT=”console”
GRUB_CMDLINE_LINUX=”console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet intel_iommu=on”
GRUB_DISABLE_RECOVERY=”true”
audit=1
First, make a backup of /etc/default/grub.
# cp -p /etc/default/grub /etc/default/grub/.$(date +%F_%R)
Edit the line below.
“GRUB_CMDLINE_LINUX=\”console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet\”
Change it to this.
“GRUB_CMDLINE_LINUX=\”console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet intel_iommu=on\”
Now, rebuild GRUB config as shown below.
# grub2-mkconfig -o /boot/grub2/grub.cfg
Specify the number of VFs to Create in rc.local
Add the following line to /etc/rc.d/rc.local, adjusting for the device name and #VFs. In this instance, our physical adapters each support 32 VFs.
echo 32 > /sys/class/net/<device>/device/sriov_numvfs
For example:
# echo 32 > /sys/class/net/p1p1/device/sriov_numvfs
# echo 32 > /sys/class/net/p3p1/device/sriov_numvfs
Also,ensure the correct SELinux context is restored.
# restorecon -R -v /etc/rc.d/rc.local
Ensure that /etc/rc.d/rc.local is executable.
#chmod +x /etc/rc.d/rc.local
Whitelist PCI devices nova-compute (Compute)
Tell nova-compute which pci devices are allowed to be passed through. Edit the file /etc/nova/nova.conf:
[default]
pci_passthrough_whitelist = [{“vendor_id”:”8086″,”product_id”:”154d”}][{“devname”:”p1p1″,”physical_network”:”sriov_net1″},{“devname”:”p3p1″,”physical_network”:”sriov_net2″}]
This tells nova that all VFs belonging to the physical interface, “p1p1“, are allowed to be passed through to VMs and belong to the neutron provider network “sriov_net1” and all VFs belonging to the physical interface, “p3p1“, are allowed to be passed through for the network “sriov_net2“.
Restart nova-compute on each compute node with the command shown below.
#systemctl restart openstack-nova-compute
Install and Enable Neutron Sriov-Agent (Compute)
Note that the sriov-agent is not required, however, we are going to install and configure it anyway.
Install the following rpm.
# yum -y install openstack-neutron-sriov-nic-agent
Now, on each compute node edit the file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini:
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver[sriov_nic]
physical_device_mappings = sriov_net1:p1p1,sriov_net2:p3p1
Now enable and start the nic agent.
# systemctl enable neutron-sriov-nic-agent.service && systemctl start neutron-sriov-nic-agent.service
Controller Node Configuration
Perform the following steps on each Controller Node. Note we will modify Nova and Neutron config files.
Neutron-Server changes in /etc/neutron/plugins/ml2/ml2_conf.ini(Controller)
The following changes take place in the file /etc/neutron/plugins/ml2/ml2_conf.ini.
Add sriovnicswitch as mechanism driver.
mechanism_drivers =openvswitch,bsn_ml2,sriovnicswitch
Set type_drivers to vlan as shown below.
type_drivers = vlan
Set tenant_network_types to vlan.
tenant_network_types = vlan
Set flat_networks as shown below where “sriov_net1” and “sriov_net2” are the networks we are going to create.
flat_networks =datacentre,sriov_net1,sriov_net2
Add VLAN ranges for the SRIOV networks to the network_vlan_ranges line as shown below.
network_vlan_ranges =datacentre:10:100,datacentre:101:122,sriov_net1:200:300,sriov_net1:200:300
Neutron-Server changes in /etc/neutron/plugins/ml2/ml2_conf_sriov.ini(Controller)
The change below needs to be made in /etc/neutron/plugins/ml2/ml2_conf_sriov.ini on each controller
Update the /etc/neutron/plugins/ml2/ml2_conf_sriov.ini on each controller.
In our case,the vendor_id is 8086 and the product_id is 10ed.
supported_pci_vendor_devs = 8086:10ed
Modify Nuetron-Server Startup
Edit /usr/lib/systemd/system/neutron-server.service. Here we add –config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini to the ExecStart line. See example below.
# cat neutron-server.service
[Unit]
Description=OpenStack Neutron Server
After=syslog.target network.target[Service]
Type=notify
User=neutron
ExecStart=/usr/bin/neutron-server –config-file /usr/share/neutron/neutron-dist.conf –config-dir /usr/share/neutron/server –config-file /etc/neutron/neutron.conf –config-file /etc/neutron/plugin.ini –config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini –config-dir /etc/neutron/conf.d/common –config-dir /etc/neutron/conf.d/neutron-server –log-file /var/log/neutron/server.log
PrivateTmp=true
NotifyAccess=all
KillMode=process[Install]
WantedBy=multi-user.target
Restart neutron on the controllers via pacemaker. See command below.
#pcs resource restart neutron-server-clone
Configure nova-scheduler (Controller)
On every controller node running nova-scheduler add PCIDeviceScheduler to the scheduler_default_filters parameter.
Also add a new line for scheduler_available_filters parameter under the [default] section in /etc/nova/nova.conf.
[DEFAULT]
scheduler_default_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter
scheduler_available_filters = nova.scheduler.filters.all_filters
scheduler_available_filters = nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter
Now restart nova-scheduler via Pacemaker as shown below.
# pcs resource restart openstack-nova-scheduler-clone
Reference
http://docs.openstack.org/liberty/networking-guide/adv-config-sriov.html