OpenStack: Mapping Ironic Hostnames to Nova Hostnames

Ironic_mascot_color

The Hostname Problem

When deploying OpenStack via Red Hat OSP director you configure the hostname of your baremetal (ironic) nodes at time of import. This is done via json file, by default named instack-env.json (but often re-named, nodes.json). Below is an excerpt from that file.

{
“nodes” :  [
{
“arch”: “x86_64”,
“cpu”: “4”,
“disk”: “40”,
“mac”: [
“58:8a:5a:e6:c0:40”
],
“memory”: “6144”,
“name”: “fatmin-ctrl0”,
“pm_addr”: “10.10.1.100”,
“pm_password”: “Mix-A-Lot”,
“pm_type”: “pxe_ipmitool”,
“pm_user”: “sir”
}

 

In the sample instance above, I am importing a node named, “fatmin-ctrl01”. This will be the server name as it appears in Ironic.  When heat deploys the overcloud, this node will by default be renamed overcloud-controller0, and any controller nodes will iterate by 1. Same situation for compute nodes.

What is preferable is to configure what is referred to as “Predictable Hostnames”. Using “Predictable Hostnames” we can do one of two things.

  1. Specify the hostname format to use and allow nova to iterate through nodes on its own.
  2. Specify the exact hostname for nova to use for each baremetal node

Nova Scheduler Hints

Before we can use either of the two options above, we must first update each baremetal node with a nova scheduler hint. In the examples below we are tagging one node to build as controller-0 (overcloud-controller0) and one node to build as (overcloud-compute-0).

For Controllers: Repeat for each controller

# ironic node-update <id> replace properties/capabilities=”node:controller-0,boot_option:local”

For Compute Node: Repeat for each compute node

# ironic node-update <id> replace properties/capabilities=”node:compute-0,boot_option:local”

You will then need to set your nova hints

parameter_defaults:
ControllerSchedulerHints:
‘capabilities:node’: ‘controller-%index%’
ComputeSchedulerHints:
‘capabilities:node’: ‘compute-%index%’

FYI – the same process can be used for the following hostname types

  • ControllerSchedulerHints
  • ComputeSchedulerHints
  • BlockStorageSchedulerHints
  • ObjectStorageSchedulerHints
  • CephStorageSchedulerHints

Custom Nova Hostname Format

Referring to option 1 shown above, we can set a specific format to be used for hostnames instead of the default.

 ControllerHostnameFormat: ‘fatmin-controller-%index%’
ComputeHostnameFormat: ‘fatmin-compute-%index%’

Using the method above the first compute node will be names fatmin-controller-01, and the first compute node will be names fatmin-compute-01. Additional nodes will iterate the index.

While this is nice, as it allows us to set a customized hostname format  for each type of node, it does not allow us to specify the exact hostname to be used for a specific ironic node.  We can do that will the HostnameMap.

HostnameMap

Now you may want to take this a bit further. You may want to use a custom nova name for each node compute/controller node. You can accomplish this using a HostnameMap as shown below.

HostnameMap:
overcloud-controller-0: fatmin-controller-0
overcloud-controller-1: fatmin-controller-1
overcloud-controller-2: fatmin-controller-2
overcloud-compute-0: fatmin-compute-0

 

Note, when specifying the flavor profiles in the deploy command for preassigned nodes, they should be specified as ‘baremetal‘ instead of ‘control‘ and ‘compute‘. This means that you will not have to assign a profile to each host. You will let the nova scheduler hints handle the decision

–control-flavor baremetal \
–compute-flavor baremetal \

So at this point – we will be able to allign the compute or controller index in ironic, with the index in Ironic. For example you can now map your ironic-node name (for example) fatmin-ctrl0 to fatmin-controller0.

Special Notes for Special People

  1. I do not suggest setting the nova name to the exactly the same name that you defined for the ironic name. While the indexes should match, the name formats should vary enough that you can easily tell if you are looking at a nova name or an ironic name.
  2. The use of HostnameMap will easily facilitate the replacement of a failed node so that you can provision the new node with the same nova name that was used by the original node before its premature death. Otherwise, nova will blacklist the nova name of the failed node. For example if controller0 dies and you need to replace and redeploy it, it will end up being named controller4 since this is the next number in the index.

 

 

OpenStack: Configuring SR-IOV in RHEL OSP 8

openstack

Introduction

This article documents the configuration used to configure SR-IOV in OSP 8/Liberty on Dell hardware

Compute Node Configuration

This section will outline the changes needed to configure SR-IOV on each Compute Node.

Bios Configuration on Dell Compute Nodes

First, you will need to ssh to the drac of each Compute Node. Then, type the command below to enter racadm command line.

#racadm

Type the command below to enable SRIOV.

#racadm set BIOS.IntegratedDevices.SriovGlobalEnable Enabled
[Key=BIOS.Setup.1-1#IntegratedDevices]
RAC1017: Successfully modified the object value and the change is in
pending state.
To apply modified value, create a configuration job and reboot
the system. To create the commit and reboot jobs, use “jobqueue”
command. For more information about the “jobqueue” command, see RACADM
help.

Type the command below to verify your configuration.

#racadm>>get BIOS.IntegratedDevices.SriovGlobalEnable

racadm get BIOS.IntegratedDevices.SriovGlobalEnable
[Key=BIOS.Setup.1-1#IntegratedDevices]
SriovGlobalEnable=Enabled

If the server already has an OS, reboot it to make these settings stick. If it doesn’t, use these racadm commands to power cycle.

#racadm serveraction powerdown
#racadm serveraction powerup

Grub Configuration on Compute Nodes

Add “intel_iommu=on” to the GRUB_CMDLINE_LINUX line as shown below.

# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=”$(sed ‘s, release .*$,,g’ /etc/system-release)”
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT=”console”
GRUB_CMDLINE_LINUX=”console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet intel_iommu=on”
GRUB_DISABLE_RECOVERY=”true”
audit=1

First, make a backup of /etc/default/grub.

# cp -p /etc/default/grub /etc/default/grub/.$(date +%F_%R)

Edit the line below.

“GRUB_CMDLINE_LINUX=\”console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet\”

Change it to this.

“GRUB_CMDLINE_LINUX=\”console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet intel_iommu=on\”

Now, rebuild GRUB config as shown below.

# grub2-mkconfig -o /boot/grub2/grub.cfg

Specify the number of VFs to Create in rc.local

Add the following line to /etc/rc.d/rc.local, adjusting for the device name and #VFs. In this instance, our physical adapters each support 32 VFs.

echo 32 > /sys/class/net/<device>/device/sriov_numvfs

For example:

# echo 32 > /sys/class/net/p1p1/device/sriov_numvfs
# echo 32 > /sys/class/net/p3p1/device/sriov_numvfs

Also,ensure the correct SELinux context is restored.

# restorecon -R -v /etc/rc.d/rc.local

Ensure that /etc/rc.d/rc.local is executable.

#chmod +x /etc/rc.d/rc.local

Whitelist PCI devices nova-compute (Compute)

Tell nova-compute which pci devices are allowed to be passed through. Edit the file /etc/nova/nova.conf:

[default]
pci_passthrough_whitelist = [{“vendor_id”:”8086″,”product_id”:”154d”}][{“devname”:”p1p1″,”physical_network”:”sriov_net1″},{“devname”:”p3p1″,”physical_network”:”sriov_net2″}]

This tells nova that all VFs belonging to the physical interface, “p1p1“, are allowed to be passed through to VMs and belong to the neutron provider network “sriov_net1” and all VFs belonging to the physical interface, “p3p1“, are allowed to be passed through for the network “sriov_net2“.

Restart nova-compute on each compute node with the command shown below.

#systemctl restart openstack-nova-compute

Install and Enable Neutron Sriov-Agent (Compute)

Note that the sriov-agent is not required, however, we are going to install and configure it anyway.

Install the following rpm.

# yum -y install openstack-neutron-sriov-nic-agent

Now, on each compute node edit the file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini:

[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

[sriov_nic]
physical_device_mappings = sriov_net1:p1p1,sriov_net2:p3p1

Now enable and start the nic agent.

# systemctl enable neutron-sriov-nic-agent.service && systemctl start neutron-sriov-nic-agent.service

Controller Node Configuration

Perform the following steps on each Controller Node. Note we will modify Nova and Neutron config files.

Neutron-Server changes in /etc/neutron/plugins/ml2/ml2_conf.ini(Controller)

The following changes take place in the file /etc/neutron/plugins/ml2/ml2_conf.ini.

Add sriovnicswitch as mechanism driver.

mechanism_drivers =openvswitch,bsn_ml2,sriovnicswitch

Set type_drivers to vlan as shown below.

type_drivers = vlan

Set tenant_network_types to vlan.

tenant_network_types = vlan

Set flat_networks as shown below where “sriov_net1” and “sriov_net2” are the networks we are going to create.

flat_networks =datacentre,sriov_net1,sriov_net2

Add VLAN ranges for the SRIOV networks to the network_vlan_ranges line as shown below.

network_vlan_ranges =datacentre:10:100,datacentre:101:122,sriov_net1:200:300,sriov_net1:200:300

Neutron-Server changes in /etc/neutron/plugins/ml2/ml2_conf_sriov.ini(Controller)

The change below needs to be made in /etc/neutron/plugins/ml2/ml2_conf_sriov.ini on each controller

Update the /etc/neutron/plugins/ml2/ml2_conf_sriov.ini on each controller.
In our case,the vendor_id is 8086 and the product_id is 10ed.

supported_pci_vendor_devs = 8086:10ed

Modify Nuetron-Server Startup

Edit /usr/lib/systemd/system/neutron-server.service. Here we add –config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini to the ExecStart line. See example below.

# cat neutron-server.service
[Unit]
Description=OpenStack Neutron Server
After=syslog.target network.target

[Service]
Type=notify
User=neutron
ExecStart=/usr/bin/neutron-server –config-file /usr/share/neutron/neutron-dist.conf –config-dir /usr/share/neutron/server –config-file /etc/neutron/neutron.conf –config-file /etc/neutron/plugin.ini –config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini –config-dir /etc/neutron/conf.d/common –config-dir /etc/neutron/conf.d/neutron-server –log-file /var/log/neutron/server.log
PrivateTmp=true
NotifyAccess=all
KillMode=process

[Install]
WantedBy=multi-user.target

Restart neutron on the controllers via pacemaker. See command below.

#pcs resource restart neutron-server-clone

Configure nova-scheduler (Controller)

On every controller node running nova-scheduler add PCIDeviceScheduler to the scheduler_default_filters parameter.

Also  add a new line for scheduler_available_filters parameter under the [default] section in /etc/nova/nova.conf.

[DEFAULT]
scheduler_default_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter
scheduler_available_filters = nova.scheduler.filters.all_filters
scheduler_available_filters = nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter

Now restart nova-scheduler via Pacemaker as shown below.

# pcs resource restart openstack-nova-scheduler-clone

 

Reference

http://docs.openstack.org/liberty/networking-guide/adv-config-sriov.html

Mapping Libvirt VM Names with OpenStack Instance Names

openstack

Within OpenStack, each virtual machine instance running on a Compute nodes also a virtual machine running on a libvirt node.

If you ssh to a Compute node and run the command below you can get the names of each VM running or registered on this Compute node.

# virsh list –all
Id Name State
—————————————————-
2 instance-000000f0 running
– instance-00000024 shut off
– instance-00000039 shut off
– instance-000000ea shut off

So there is only one VM currently running on this Compute node, but which VM is it?

Well, we can figure that out pretty easily. See below.

# virsh dumpxml instance-000000f0 | grep uuid | grep name

<entry name=’uuid’>3103d38c-447d-40af-9607-56b26473ee72</entry>

Now we just have to map this UUID back to an OpenStack instance name

Here we have a nasty little awk grep to get the UUID and name of each OpenStack instance running in our cluster.

# nova list | grep -v “+” | grep -v ID | awk ‘{print $2 $3 $4}’

a402716b-73d0-4303-9331-202bc2386ab8|storage-perf-a
bd73f092-88c9-4af1-b569-c1176290841c|storage-perf-b
0ab8decb-6623-4e68-b8e4-b9cd522f6ea9|storage-perf-c
493c8afe-836e-4846-bdd0-029bb6e7f70e|storage-perf-d

Note that you can also get the instance name using a UUID,  see below

#nova show db79f6a2-455e-4f17-88d0-b3018d279c7c | grep instanceOS-EXT-SRV-ATTR:instance_name | instance-0000002a

OS-EXT-SRV-ATTR:instance_name | instance-0000002a

 

Update:

Thanks to TB for showing me this one. You can accomplish all of this via nova cli using the FQDN of the host.

(overcloud) [stack@undercloud ~]$ nova list --all --host comp39.localdomain --fields=name,host,instance_name,status
+--------------------------------------+----------------------+-------------------------------+-------------------+--------+
| ID | Name | Host | Instance Name | Status |
+--------------------------------------+----------------------+-------------------------------+-------------------+--------+
| 30fccd83-e517-4e56-a1df-5d5d454818ff | vm-name1 | comp39.localdomain | instance-0023a423 | ACTIVE |
| bc6bb4da-1cac-4603-ad52-2078f13cf6fd | vm-name-2 | comp39.localdomain | instance-0023a348 | ACTIVE |
+--------------------------------------+----------------------+-------------------------------+-------------------+--------+

 

 

Stack_Stat: A Simple OpenStack Service Checking Function

openstack

Looking for a simple way to see what OpenStack services are enabled, running, or dead on an OpenStack Compute or Controller Node, and also want to see when these services were started? Then check out this simple function that you can add to root’s .bashrc. Note that I do not take credit for writing this (did modify it a tiny bit, however), rather it is something a friend of mine passed on to me and I found it too handy not to document and share.

First append the following to /root/.bashrc.

[code language=”css”]
##Stack Stat Function
function stack_stat {
services=$( systemctl list-unit-files | egrep ‘(openstack|neutron)’ |awk ‘{print $1}’)
for service in $services ;do
echo -n "$service: "
systemctl status $service |grep Active
done
}
[/code]

What this function does is show you the status or “Active” (or enabled as you would traditionally say) OpenStack Services. Just run the following…

#stack_stat

You will see something similar to the output below, which shows you each enabled OpenStack service and its running status.

# stack_stat
neutron-dhcp-agent.service: Active: active (running) since Fri 2015-07-10 09:35:19 EDT; 3 days ago
neutron-l3-agent.service: Active: active (running) since Fri 2015-07-10 09:35:19 EDT; 3 days ago
neutron-lbaas-agent.service: Active: active (running) since Fri 2015-07-10 09:35:17 EDT; 3 days ago
neutron-metadata-agent.service: Active: active (running) since Fri 2015-07-10 09:35:17 EDT; 3 days ago
neutron-metering-agent.service: Active: active (running) since Fri 2015-07-10 09:35:17 EDT; 3 days ago
neutron-netns-cleanup.service: Active: inactive (dead)
neutron-openvswitch-agent.service: Active: active (running) since Fri 2015-07-10 09:35:19 EDT; 3 days ago
neutron-ovs-cleanup.service: Active: active (exited) since Fri 2015-07-10 09:35:19 EDT; 3 days ago
neutron-server.service: Active: active (running) since Fri 2015-07-10 09:35:29 EDT; 3 days ago
openstack-ceilometer-alarm-evaluator.service: Active: active (running) since Fri 2015-07-10 09:35:17 EDT; 3 days ago
openstack-ceilometer-alarm-notifier.service: Active: active (running) since Fri 2015-07-10 09:35:17 EDT; 3 days ago
openstack-ceilometer-api.service: Active: active (running) since Fri 2015-07-10 09:35:17 EDT; 3 days ago
openstack-ceilometer-central.service: Active: active (running) since Fri 2015-07-10 09:35:17 EDT; 3 days ago

Now compare the output above to the output of “openstack-service status” as shown below. Note that you do not get time and date stamps.

# openstack-service status
neutron-dhcp-agent (pid 6725) is active
neutron-l3-agent (pid 6724) is active
neutron-lbaas-agent (pid 3884) is active
neutron-metadata-agent (pid 3886) is active
neutron-metering-agent (pid 3882) is active
neutron-openvswitch-agent (pid 6726) is active
neutron-server (pid 3885) is active
openstack-ceilometer-alarm-evaluator (pid 3899) is active
openstack-ceilometer-alarm-notifier (pid 3897) is active
openstack-ceilometer-api (pid 3898) is active
openstack-ceilometer-central (pid 3895) is active
openstack-ceilometer-collector (pid 3893) is active
openstack-ceilometer-notification (pid 3890) is active

I know not a huge difference, but it’s still somewhat handy.

OpenStack Nova: Get NoVNC Connection Info From the Command Line

vampirebatTo access a novnc console in OpenStack without logging into the Horizon WebUi, follow the steps below.

First and foremost you need to find your virtual instance. In this example I am looking for one of my test instances.

# nova list | grep test

Note that if you are logged in as root and the instance you are looking for was created under a guest tenant, then the command is a bit different. See below.

# nova list –all-tenants | grep test

The nova list command above can spew out a lot of information if you have many test instances. However this is the one that I am looking for specifically.

2a5481e6-9210-4353-af8f-db49e052d5c6 | fatmin-test-1         | ACTIVE  | –          | Running     | fatmin_network=10.0.10.1

Now grab the UUID from the first field above.

# nova get-vnc-console 2a5481e6-9210-4353-af8f-db49e052d5c6 novnc

This will spit out a URL as shown below.

+——-+———————————————————————————–+
| Type | Url |
+——-+———————————————————————————–+
| novnc | http://10.21.148.95:6080/vnc_auto.html?token=f27af916-94a0-4a96-85fd-7b74b0748ce1 |
+——-+———————————————————————————–+

Now copy and paste this into a browser window to access the console for your instance.