OpenStack: Deleting Zombie Cinder Volumes and VMs

cinder-1

First off let me start by saying that the new Cinder logo is wonderful. Nothing helps me think of backend storage better than the backend of a horse.

In an environment I am working in, we have a large number of cinder volumes that are in error state, due to the backend storage being ripped out. The volumes were not deleted, nor were they detached from the VMs.

End result, you cannot delete the zombie VM (at it has an attached volume) and you cannot delete the zombie/orphaned volume (as it is attached to a VM).

The following process allows you to work around the chicken-and-egg scenario above.

First we get a list of all volumes in error state.

# openstack volume list –all | grep -i error

Then we take a closer look at the volume to see if it exists/existed on the backend that was removed.

# openstack volume show 05b372ef-ee45-499b-9676-72cc4170e1b3

First we check the backend to ensure it is the affected backend – in this case it is.

| os-vol-host-attr:host | hostgroup@dellsc#openstack_dellsc

We also check for any current attachments. Below we see that this volume is attached to a vm with the uuid shown below.

| attachments | [{u’server_id’: u’d142eb4b-823d-4abd-95a0-3b02a3194c9f’,

Now we reset the state of the volume, so that it is no longer in an error state

# cinder reset-state –state available 05b372ef-ee45-499b-9676-72cc4170e1b3

Now we detach the volume via cinder.

# cinder reset-state –attach-status detached 05b372ef-ee45-499b-9676-72cc4170e1b3

Now we are free to delete the volume

# openstack volume delete 05b372ef-ee45-499b-9676-72cc4170e1b3

Confirm volume deletion

# openstack volume show 05b372ef-ee45-499b-9676-72cc4170e1b3
No volume with a name or ID of ’05b372ef-ee45-499b-9676-72cc4170e1b3′ exists

Now we can delete the VM.

# openstack server delete d142eb4b-823d-4abd-95a0-3b02a3194c9f

And now we confirm its deletion.

#openstack server show d142eb4b-823d-4abd-95a0-3b02a3194c9f
No server with a name or ID of ‘d142eb4b-823d-4abd-95a0-3b02a3194c9f’ exists.

OpenStack Post Install Validation Checklist

smarta

Below is a “Post Install Validation Checklist” that I am working on for validating an OpenStack Deployment. This is a work in progress and is sure to be missing some items. I plan to attempt to treat this post as a live document an update as I find new items to add to the checklist

Component Task
All On each node (compute, controller) apply post-install configurations. Ensure ssh keys for root and nova exist on all hosts
All If deploying with Staypuft installer, ensure that puppet is stopped and disabled so that it does not overwrite post-install configurations
Glance Ensure that you can upload and configure OS Images
Nova Ensure that you can create flavors
Nova Ensure that you can successfully provision test instances. Test with each flavor and OS Image
Cinder Provision and assign a block device from Cinder. Also ensure that you can detach and delete a block device
Neutron Create and test tenant networks
Neutron Provision and assign a floating IP from an external network for each tenant network. Ensure that connectivity is successful
Nova Define and assign a security group to an instance, enabling at least SSH and ICMP access
All Reboot of all nodes to ensure configuration persistence, document any configuration changes. Push any fixes out to nodes when necessary
Nova Generate and test keypairs to be used for access to instances
Nova Ensure you are able to create and delete instance snapshots
Swift If using swift, verify that you can create object stor and upload or create files
Nova Test Live Migration by migrating instances across each compute node