Introduction
This error you can see in multiple places, one being the logs for rabbitmq. The second being /var/log/cinder/volume.log
The tail statement below is very helpful for finding errors in OpenStack.
tail -fn0 /var/log/{nova,cinder,glance}/*.log | egrep 'ERROR|TRACE|WARNING'
Documented Errors:
EXAMPLE 1:
oslo.messaging._drivers.impl_rabbit PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - cannot redeclare exchange 'openstack' in vhost '/' with different type, durable, internal or autodelete value
EXAMPLE 2:
2015-06-08 09:52:17.367 8437 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to declare consumer for topic 'cinder-scheduler': Queue.declare: (406) PRECONDITION_FAILED - parameters for queue 'cinder-scheduler' in vhost '/' not equivalent
Warning
Note that you need to make sure that you are only working with OpenStack specific queues.
Cause
This is caused by a mismatch in the durable queues setting in the configuration files on the controller nodes. You cannot just change this setting to correct the issue, you must manually delete the affected queues.
Resolution
# pcs resource unmanage mysqld # pcs resource unmanage rabbitmq-server # pcs cluster standby --all # curl http://localhost:15672/cli/rabbitmqadmin > rabbitmqadmin # chmod +x rabbitmqadmin # ./rabbitmqadmin help subcommands # rabbitmqctl list_queues
The ccommand above will give you a list of all the queues, so now you can search for the affected queue and delete it as shown. In this example I am deleting the queue “notifications.info”.
[root@lppcldiuctl01 ~(openstack_admin)]# ./rabbitmqadmin --username=openstack --password=81d86697132a45a55 delete queue name=notifications.info queue deleted
If you have multiple queues affected you can run through several at a time as shown below.
# rabbitmqctl list_queues | awk '{print $1}' > queues # vim queues # cat /etc/rabbitmq/rabbitmq.config # ./rabbitmqadmin --username=openstack --password=81d86697132a45d9832d7fb35d168a55 delete queue name=reply_ea13ffaf100de1baca # for q in $(<queues); do ./rabbitmqadmin --username=openstack --password=81d8669713 delete queue name=$q; done
Now bring everything back online
# pcs cluster unstandby --all
Now re-managed the services that you unmanaged above.
# pcs resource manage mysqld # pcs resource manage rabbitmq-server
Note that the documentation above can be modified to delete exchanges as well as queues. More information on exchanges below.