In this post we are going to walk through a process you can use to troubleshoot a failed heat stack deploy by viewing information of nested resources.
In the command below we are looking for any failed resource. Our grep allows us to grab the name of Parent id.
[stack@undercloud] # heat resource-list -n5 overcloud | grep -A5 -B5 -i Failed
What is below its very hard to read as I am unable to get the copy/paste to format correctly. What we are looking for is the parent resource for the failed child resource. I have bolded this line below to make is easier to find.
| StoragePort | ed706420-e4e2-4e46-b3d3-8dfd47855a3a | OS::Neutron::Port | CREATE_COMPLETE | 2016-03-03T18:48:42Z | StorageVirtualIP |
| VipPort | 2833b468-5f0f-4162-a0bb-e928126b3767 | OS::Neutron::Port | CREATE_COMPLETE | 2016-03-03T18:48:42Z | RedisVirtualIP |
| 0 | 6c8b39f8-4ed2-47e4-957b-f74b8520d386 | OS::TripleO::Compute | CREATE_IN_PROGRESS | 2016-03-03T18:48:47Z | Compute |
| 0 | 829a6b13-7a92-483a-b9d3-52a99dc655f8 | OS::TripleO::Controller | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z | Controller |
| 1 | 556da671-baa6-4fc8-be83-f3acfb324f46 | OS::TripleO::Controller | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z | Controller |
| 2 | c28d5942-a206-4734-bfd1-c6d263fafc52 | OS::TripleO::Controller | CREATE_FAILED | 2016-03-03T18:48:49Z | Controller |
| InternalApiPort | 59800552-0b66-4437-bf43-833df8131673 | OS::TripleO::Compute::Ports::InternalApiPort | CREATE_COMPLETE | 2016-03-03T18:48:49Z | 0 |
| NetIpMap | 2d455861-3cbe-40cd-8443-f0e63b0c24b7 | OS::TripleO::Network::Ports::NetIpMap | CREATE_COMPLETE | 2016-03-03T18:48:49Z | 0 |
| NetworkConfig | 062eb0ed-7914-435b-8f1e-61395fed6548 | OS::TripleO::Compute::Net::SoftwareConfig | CREATE_COMPLETE | 2016-03-03T18:48:49Z | 0 |
| NetworkDeployment | 5a11128b-c62a-4720-a8b2-6f27b0d83d74 | OS::TripleO::SoftwareDeployment | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z | 0 |
| NodeUserData | 94deb049-9015-490a-9a2b-925ff17c5c9f | OS::TripleO::NodeUserData | CREATE_COMPLETE | 2016-03-03T18:48:49Z | 0 |
Our parent resource is “Controller” so let’s take a look at that first. We want to grab the “physical_resource_id“. Again, hard to see so I have bolded that line below.
[stack@undercloud] # heat resource-show overcloud Controller
+————————+————————————-+
| Property | Value |
+————————+—————————–+
| attributes | { |
| | “attributes”: null, |
| | “refs”: null |
| | } |
| description | |
| links | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud/054fde22-87f2-44cb-8318-784d1fa46323/resources/Controller (self) |
| | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud/054fde22-87f2-44cb-8318-784d1fa46323 (stack) |
| | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud-Controller-swdfj36bsf6e/6faa959c-3f1b-4f60-aeee-c01e34f399e7 (nested) |
| logical_resource_id | Controller |
| physical_resource_id | 6faa959c-3f1b-4f60-aeee-c01e34f399e7 |
| required_by | ControllerNodesPostDeployment |
| | VipDeployment |
| | ControllerIpListMap |
| | ControllerBootstrapNodeDeployment |
| | ControllerClusterDeployment |
| | CephClusterConfig |
| | ControllerSwiftDeployment |
| | SwiftDevicesAndProxyConfig |
| | ControllerBootstrapNodeConfig |
| | ControllerCephDeployment |
| | allNodesConfig |
| | ControllerAllNodesDeployment |
| | ControllerClusterConfig |
| resource_name | Controller |
| resource_status | CREATE_IN_PROGRESS |
| resource_status_reason | state changed |
| resource_type | OS::Heat::ResourceGroup |
| updated_time | 2016-03-03T18:48:34Z |
+————————+————————————–+
Now that we know the physical_resource_id we can drill down into that resource. Note resource “2” is the one we want to focus on.
[stack@undercloud] # heat resource-list 6faa959c-3f1b-4f60-aeee-c01e34f399e7
+—————+—————————+———————-+
| resource_name | physical_resource_id | resource_type | resource_status | updated_time |
+—————+————————+———————-+
| 0 | 829a6b13-7a92-483a-b9d3-52a99dc655f8 | OS::TripleO::Controller | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z |
| 1 | 556da671-baa6-4fc8-be83-f3acfb324f46 | OS::TripleO::Controller | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z |
| 2 | c28d5942-a206-4734-bfd1-c6d263fafc52 | OS::TripleO::Controller | CREATE_FAILED | 2016-03-03T18:48:49Z |
+—————+———————————————–+
Now we run “heat resource-show” using the physical resource id plus the resource_name (which is 2).
[stack@undercloud] # heat resource-show 6faa959c-3f1b-4f60-aeee-c01e34f399e7 2
+————————+——————————————–+
| Property | Value |
+————————+——————————————–+
| attributes | { |
| | “storage_mgmt_ip_address”: null, |
| | “hostname”: “overcloud-controller-2”, |
| | “config_identifier”: “,”, |
| | “nova_server_resource”: “52872fac-9db2-4913-8e02-281fc01f44eb”, |
| | “tenant_ip_address”: null, |
| | “external_ip_address”: null, |
| | “swift_device”: “r1z1-:%PORT%/d1”, |
| | “corosync_node”: { |
| | “ip”: “172.16.0.136”, |
| | “name”: “overcloud-controller-2” |
| | }, |
| | “hosts_entry”: ” overcloud-controller-2.localdomain overcloud-controller-2 overcloud”, |
| | “swift_proxy_memcache”: “:11211”, |
| | “storage_ip_address”: null, |
| | “internal_api_ip_address”: null, |
| | “ip_address”: “172.16.0.136” |
| | } |
| description | |
| links | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud-Controller-swdfj36bsf6e/6faa959c-3f1b-4f60-aeee-c01e34f399e7/resources/2 (self) |
| | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud-Controller-swdfj36bsf6e/6faa959c-3f1b-4f60-aeee-c01e34f399e7 (stack) |
| | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud-Controller-swdfj36bsf6e-2-j5zi5fahlwf3/c28d5942-a206-4734-bfd1-c6d263fafc52 (nested) |
| logical_resource_id | 2 |
| parent_resource | Controller |
| physical_resource_id | c28d5942-a206-4734-bfd1-c6d263fafc52 |
| required_by | |
| resource_name | 2 |
| resource_status | CREATE_FAILED |
| resource_status_reason | CREATE aborted |
| resource_type | OS::TripleO::Controller |
| updated_time | 2016-03-03T18:48:49Z |
+————————+——————————————————————————————————————–+