First off let me start by saying that the new Cinder logo is wonderful. Nothing helps me think of backend storage better than the backend of a horse.
In an environment I am working in, we have a large number of cinder volumes that are in error state, due to the backend storage being ripped out. The volumes were not deleted, nor were they detached from the VMs.
End result, you cannot delete the zombie VM (at it has an attached volume) and you cannot delete the zombie/orphaned volume (as it is attached to a VM).
The following process allows you to work around the chicken-and-egg scenario above.
First we get a list of all volumes in error state.
# openstack volume list –all | grep -i error
Then we take a closer look at the volume to see if it exists/existed on the backend that was removed.
# openstack volume show 05b372ef-ee45-499b-9676-72cc4170e1b3
First we check the backend to ensure it is the affected backend – in this case it is.
When deploying OpenStack via Red Hat OSP director you configure the hostname of your baremetal (ironic) nodes at time of import. This is done via json file, by default named instack-env.json (but often re-named, nodes.json). Below is an excerpt from that file.
In the sample instance above, I am importing a node named, “fatmin-ctrl01”. This will be the server name as it appears in Ironic. When heat deploys the overcloud, this node will by default be renamed overcloud-controller0, and any controller nodes will iterate by 1. Same situation for compute nodes.
What is preferable is to configure what is referred to as “Predictable Hostnames”. Using “Predictable Hostnames” we can do one of two things.
Specify the hostname format to use and allow nova to iterate through nodes on its own.
Specify the exact hostname for nova to use for each baremetal node
Nova Scheduler Hints
Before we can use either of the two options above, we must first update each baremetal node with a nova scheduler hint. In the examples below we are tagging one node to build as controller-0 (overcloud-controller0) and one node to build as (overcloud-compute-0).
Using the method above the first compute node will be names fatmin-controller-01, and the first compute node will be names fatmin-compute-01. Additional nodes will iterate the index.
While this is nice, as it allows us to set a customized hostname format for each type of node, it does not allow us to specify the exact hostname to be used for a specific ironic node. We can do that will the HostnameMap.
Now you may want to take this a bit further. You may want to use a custom nova name for each node compute/controller node. You can accomplish this using a HostnameMap as shown below.
Note, when specifying the flavor profiles in the deploy command for preassigned nodes, they should be specified as ‘baremetal‘ instead of ‘control‘ and ‘compute‘. This means that you will not have to assign a profile to each host. You will let the nova scheduler hints handle the decision
So at this point – we will be able to allign the compute or controller index in ironic, with the index in Ironic. For example you can now map your ironic-node name (for example) fatmin-ctrl0 to fatmin-controller0.
Special Notes for Special People
I do not suggest setting the nova name to the exactly the same name that you defined for the ironic name. While the indexes should match, the name formats should vary enough that you can easily tell if you are looking at a nova name or an ironic name.
The use of HostnameMap will easily facilitate the replacement of a failed node so that you can provision the new node with the same nova name that was used by the original node before its premature death. Otherwise, nova will blacklist the nova name of the failed node. For example if controller0 dies and you need to replace and redeploy it, it will end up being named controller4 since this is the next number in the index.
os-collect-config is a tool that starts up via systemd when a system boots. It initially runs at boot time, but continues to run looking for changes in heat metadata. In a nutshell, os-collect-config is responsible for monitoring and downloading metadata from the Heat API.
When data changes, os-collect-config makes a call to os-refresh-config. This data provides the node with all of the information it needs to make configuration changes to the host – this data will be node specific.
Os-collect-config polls for data from sources (nova-metadata and heat) and stores them in /var/lib/os-collect-config/.
os-refresh-config is called by os-collect-config once it recognizes that metadata has changed within the Heat API for that specific node.
The important steps that os-refresh-config take are shown below
1) Apply systemctl configurables
2) Run os-apply-config (see below)
3) Configure the networking for the host
4) Download and set the hieradata files for puppet parameters
5) Configure /etc/hosts
6) Deploy software configuration with Heat
os-apply-config is called by os-refresh-config to sets up configuration files on specific nodes. It is called via ‘/usr/libexec/os-refresh-config/configure.d/20-os-apply-config‘.
As is does with the undercloud deploy, os-refresh-config executes scripts under /usr/libexec/os-refresh-config/ in a specific order based on numbering.
First scripts within thje pre-configure.d/ directory are run, then configure.d/ scripts are applied, and finally scripts in post-configure.d/.
It is within these scripts that the metadata downloaded by os-collect-config will be acted upon.
Any call to os-apply-config uses the files in /var/lib/os-collect-config as its configuration source.
The appropriate script files for doing so are as follows:
overcloud$ ll /usr/libexec/os-refresh-config/configure.d/
-rwxr-xr-x. 1 root root 396 Aug 5 07:31 10-sysctl-apply-config
-rwxr-xr-x. 1 root root 42 Aug 5 07:31 20-os-apply-config
-rwxr-xr-x. 1 root root 189 Aug 5 07:31 20-os-net-config
-rwxr-xr-x. 1 root root 629 Aug 5 07:31 25-set-network-gateway
-rwxr-xr-x. 1 root root 2265 Aug 5 07:31 40-hiera-datafiles
-rwxr-xr-x. 1 root root 1387 Aug 5 07:31 51-hosts
-rwxr-xr-x. 1 root root 5784 Aug 5 07:31 55-heat-config
If we run os-apply-config manually, we can see that it does the following:
The directory /os/net-config/ holds the config.json file that is used to modify the networking configuration on each host. The config found in this file is derived from the os-collect-config data in /var/lib/os-collect-config/.
Again you can use this file to review your networking configuration and compare and contrast to your templates. Formatting is off in the file below, but you get the point.