Contain Yourself
With the release of Red Hat OpenStack 13, the move to containerized overcloud services is complete. Traditional systemd services such as RabbitMQ, Haproxy, Mariadb, etc, are all now running as containers in the overcloud. This move to containers is meant to provide additional stability, control, and security to the platform. Future upgrades should be easier, and future deploys should be more flexible.
However, the move to containers brings with it a couple of new challenges. Operations.
The average OpenStack administrator no longer restarts services, they restart containers.
They no longer view a rabbit cluster’s status on the controller node, but rather within a container on the controller node. Log locations have changed. Config file locations have changed.
So let’s retrain ourselves.
Logs and Config Files
So this one is not that hard. Logs for all containerized services can be found in the directory below.
# /var/log/containers/
Service configuration files are a bit harder to find, and there are several directories worth noting.
Initial configuration write occurs in the directory below. The directory for a given service containers a full copy of configuration directories (/etc, /var/www, /root, etc).
# /var/lib/config-data/
The directory shown below contains the service config files that were modified via puppet. Directories within this tree should are copied into the running container config by kolla. If you want to modify the config of a container, do it here.
# /var/lib/config-data/puppet-generated/
Paunch
Paunch is a “utility to launch and manage containers using YAML based configuration data”. These files can be found in the directory shown below.
/var/lib/tripleo-config/
In this directory you will find docker container startup files. Example below.
[simterm]# ls | grep step_1
docker-container-startup-config-step_1.json
hashed-docker-container-startup-config-step_1.json
[/simterm]
The “non-hashed” files are input files for their “hashed” counterparts. The “hashed” file contains a TRIPLEO_CONFIG_HASH, which is an md5 sum of the config_volumes used by each individual container. See example below.
“TRIPLEO_CONFIG_HASH=f0cc3f9fc4b80c5aa1cb73bf4dc84100”
The TRIPLEO_CONFIG_HASH is used to notify paunch to restart a container if a config used in by the container was changed.
Using the CLI: View/Inspect Containers
The command below lists all running containers. The output will show you the container id, the container image used, container health, and the container name.
[simterm]# docker ps
[/simterm]
To view a list of all running or stopped containers use the command below.
[simterm]# docker ps –all
[/simterm]
List docker images using the command below
[simterm]# docker images
[/simterm]
Inspect a container with the command below. Among other info, the output will show you the bind mounts for the container
[simterm]# docker inspect
[/simterm]
Example
[simterm]# docker inspect neutron_ovs_agent
[/simterm]
The command below will show you the name of a running container. In this example we are looking for the name of the rabbit container. Here the “-f” option is used to specify a filter
[simterm]# docker ps -f name=rabbit
[/simterm]
Similary, you can query all running containers in a “healthy” state. The -q option will return only the container id.
[simterm]#docker ps -q -f health=healthy
[/simterm]
Using the CLI: Restarting Containers
Using the examples below you and start, stop, and restart containers.
[simterm]
$ sudo docker restart horizon
$ sudo docker stop horizon
$ sudo docker start horizon
[/simterm]
Using the CLI: docker exec
You can use the “docker exec” command to login via bash to a container. Example below.
[simterm]# docker exec -it redis-bundle-docker-0 /bin/bash
[/simterm]
You can also use the “docker exec” command to run a command in a container. In the example below we are running docker exec to query the rabbit cluster status in the running rabbitmq container.
[simterm]# docker exec $(docker ps -q -f name=rabbitmq) rabbitmqctl cluster_status`
[/simterm]
Pacemaker and Bundles
In version 1.1.17, Pacemaker introduced a new type of resource: the “bundle“.
A bundle is a single resource specifying the settings, networking
requirements, and storage requirements for any number of containers
generated from the same container image.
See output below from “pcs status”
[simterm]Docker container set: rabbitmq-bundle [registry.access.redhat.com/rhosp13/openstack-rabbitmq:pcmklatest]
rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started pod4-overcloud-controller-0
rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started pod4-overcloud-controller-1
rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started pod4-overcloud-controller-2
[/simterm]
The example output above shows the three containers that make up the rabbitmq bundle. One container per OpenStack controller.
Note, the exact name of the container differs from the name shown via PCS. Here the exact container name is rabbitmq-bundle-docker-0
[simterm]# docker ps | grep rabbitmq-bundle
0b92ca5ac66a registry.access.redhat.com/rhosp13/openstack-rabbitmq:pcmklatest “/bin/bash /usr/lo…” 46 hours ago Up 46 hours rabbitmq-bundle-docker-0
[/simterm]
Mapping Systemd Services to Containers
The table below is taken straight from the official documentation. [1]. The left column contains the service name, the middle the list of systemd services for that service. The column on the right lists the docker container that has replaced that systemd service.
| OpenStack Service | Systemd Services | Docker Containers |
| aodh | openstack-aodh-evaluator
openstack-aodh-listener openstack-aodh-notifier httpd (openstack-aodh-api) |
aodh_listener
aodh_api aodh_notifier aodh_evaluator |
| ceilometer | openstack-ceilometer-central
openstack-ceilometer-collector openstack-ceilometer-notification httpd (openstack-ceilometer-api) |
ceilometer_agent_notification
ceilometer_agent_central |
| cinder | openstack-cinder-api
openstack-cinder-scheduler openstack-cinder-volume |
cinder_scheduler
cinder_api openstack-cinder-volume-docker-0 |
| glance | openstack-glance-api
openstack-glance-registry |
glance_api |
| gnocchi | openstack-gnocchi-metricd
openstack-gnocchi-statsd httpd (openstack-gnocchi-api) |
gnocchi_statsd
gnocchi_api gnocchi_metricd |
| heat | openstack-heat-api-cfn
openstack-heat-api-cloudwatch openstack-heat-api openstack-heat-engine |
heat_api_cfn
heat_engine heat_api |
| horizon | httpd (openstack-dashboard) | horizon |
| keystone | httpd (openstack-keystone) | keystone |
| neutron | neutron-dhcp-agent
neutron-l3-agent neutron-metadata-agent neutron-openvswitch-agent neutron-server |
neutron_ovs_agent
neutron_l3_agent neutron_metadata_agent neutron_dhcp neutron_api |
| nova | openstack-nova-api
openstack-nova-conductor openstack-nova-consoleauth openstack-nova-novncproxy openstack-nova-scheduler |
nova_metadata
nova_api nova_conductor nova_vnc_proxy nova_consoleauth nova_api_cron nova_scheduler nova_placement |
| panko | panko_api | |
| swift | openstack-swift-account-auditor
openstack-swift-account-reaper openstack-swift-account-replicator openstack-swift-account openstack-swift-container-auditor openstack-swift-container-replicator openstack-swift-container-updater openstack-swift-container openstack-swift-object-auditor openstack-swift-object-expirer openstack-swift-object-replicator openstack-swift-object-updater openstack-swift-object openstack-swift-proxy |
swift_proxy
swift_account_server swift_container_auditor swift_object_expirer swift_object_updater swift_container_replicator swift_account_auditor swift_object_replicator swift_container_server swift_rsync swift_account_reaper swift_account_replicator swift_object_auditor swift_object_server swift_container_update |
References
https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/tips_tricks.html
https://access.redhat.com/solutions/3518521