When you Nova boot an instance in an OSP Overcloud deployed using SR-IOV and CPU Pinning, that instance will be NUMA aligned, meaning that its vCPUs, memory, and SR-IOV VF (Virtual Function) will all be local to the same NUMA node. Nova will not allow you to deploy a non-NUMA aligned instance in such an environment (which you might want to do when testing the cross-NUMA penalty of your hardware).
If you are looking to misalign an instance for the purpose of testing, you can use the following process.
First, ssh to the Compute node running your test instance. In the example below, we can see that the libvirt shows VM three as being pinned to odd numbered cores. On this hardware that means, NUMA node 1.
In this example, our VM has 4 cores.
# virsh vcpupin 3VCPU: CPU Affinity——————————0: 111: 352: 173: 41
Using “virsh vcpupin“ you can move the pinned VM cores to the pCPUs local to numa node 0.
In the example below, we start my migrating vCPU 0 to pCPU 12.
# virsh vcpupin 3 0 12
Now we migrate vCPU 1 to pCPU 14.
# virsh vcpupin 3 1 14
Now we migrate vCPU 2 to pCPU 18.
# virsh vcpupin 3 2 18
Now we migrate vCPU 3 to pCPU 20.
# virsh vcpupin 3 3 20
Now, pinning now appears as follows.
# virsh vcpupin 3VCPU: CPU Affinity———————————-0: 121: 142: 183: 20