With the release of Red Hat OpenStack 13, the move to containerized overcloud services is complete. Traditional systemd services such as RabbitMQ, Haproxy, Mariadb, etc, are all now running as containers in the overcloud. This move to containers is meant to provide additional stability, control, and security to the platform. Future upgrades should be easier, and future deploys should be more flexible.
However, the move to containers brings with it a couple of new challenges. Operations.
The average OpenStack administrator no longer restarts services, they restart containers.
They no longer view a rabbit cluster’s status on the controller node, but rather within a container on the controller node. Log locations have changed. Config file locations have changed.
So let’s retrain ourselves.
Logs and Config Files
So this one is not that hard. Logs for all containerized services can be found in the directory below.
Service configuration files are a bit harder to find, and there are several directories worth noting.
Initial configuration write occurs in the directory below. The directory for a given service containers a full copy of configuration directories (/etc, /var/www, /root, etc).
The directory shown below contains the service config files that were modified via puppet. Directories within this tree should are copied into the running container config by kolla. If you want to modify the config of a container, do it here.
Paunch is a “utility to launch and manage containers using YAML based configuration data”. These files can be found in the directory shown below.
In this directory you will find docker container startup files. Example below.
The “non-hashed” files are input files for their “hashed” counterparts. The “hashed” file contains a TRIPLEO_CONFIG_HASH, which is an md5 sum of the config_volumes used by each individual container. See example below.
The TRIPLEO_CONFIG_HASH is used to notify paunch to restart a container if a config used in by the container was changed.
Using the CLI: View/Inspect Containers
The command below lists all running containers. The output will show you the container id, the container image used, container health, and the container name.
To view a list of all running or stopped containers use the command below.
List docker images using the command below
Inspect a container with the command below. Among other info, the output will show you the bind mounts for the container
The command below will show you the name of a running container. In this example we are looking for the name of the rabbit container. Here the “-f” option is used to specify a filter
Similary, you can query all running containers in a “healthy” state. The -q option will return only the container id.
Using the CLI: Restarting Containers
Using the examples below you and start, stop, and restart containers.
Using the CLI: docker exec
You can use the “docker exec” command to login via bash to a container. Example below.
You can also use the “docker exec” command to run a command in a container. In the example below we are running docker exec to query the rabbit cluster status in the running rabbitmq container.
Pacemaker and Bundles
In version 1.1.17, Pacemaker introduced a new type of resource: the “bundle“.
A bundle is a single resource specifying the settings, networking
requirements, and storage requirements for any number of containers
generated from the same container image.
See output below from “pcs status”
The example output above shows the three containers that make up the rabbitmq bundle. One container per OpenStack controller.
Note, the exact name of the container differs from the name shown via PCS. Here the exact container name is rabbitmq-bundle-docker-0
Mapping Systemd Services to Containers
The table below is taken straight from the official documentation. . The left column contains the service name, the middle the list of systemd services for that service. The column on the right lists the docker container that has replaced that systemd service.
|OpenStack Service||Systemd Services||Docker Containers|