Ran into issues with a deploy via RHEL OSP director, which caused our heat stack scale compute scale-out to fail.

We corrected the issue, and then attempted our deploy again. This time around, we were able to scale out by several compute nodes. However, several nodes failed to deploy properly.

After trudging through the logs for a bit we were able to find the error below in /var/log/nova-conductor.log

2016-07-25 18:34:39.453 27374 ERROR nova.scheduler.utils [req-1caa3ca0-9e13-4340-b8b2-80b1cf2f8a7f 8408462bd5a8445c9742ea4dfbc20d70 cc44dc9e68064e64899697ac610c8f06 – – -] [instance: e44e3a04-a47e-4a8e-9eb5-0037c1175e4d] Error from last host: fatmin.lab.localdomain (node c5177fbf-ae0f-49db-94f4-087537b3dd53): [u’Traceback (most recent call last):\n’, u’ File “/usr/lib/python2.7/site-packages/nova/compute/manager.py”, line 1905, in _do_build_and_run_instance\n filter_properties)\n’, u’ File “/usr/lib/python2.7/site-packages/nova/compute/manager.py”, line 2058, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n’, u’RescheduledException: Build of instance e44e3a04-a47e-4a8e-9eb5-0037c1175e4d was re-scheduled: Port 14:18:77:3e:1a:bf is still in use.\n’]

To resolve we ran the following command, using the MAC address shown above to narrow down the search.

#neutron port-list | grep “14:18:77:3e:1a:bf”
| 25a366df-1e6c-4eb8-853c-5b7db82637f0 | | 14:18:77:3e:1a:bf | {“subnet_id”: “0310f210-63ad-4616-9338-d59ac13cc0be”, “ip_address”: “10.20.0.134”} |

The command “neutron port show“, showed us that this port was down and was not responding to ping.

We then deleted the port via neutron

#neutron port-delete 25a366df-1e6c-4eb8-853c-5b7db82637f0

We then re-ran our deploy and were able to scale without issue.