Red Hat OpenStack 13: Containerized Services Operations Guide

Screenshot from 2018-11-10 18-59-16.png

Contain Yourself

With the release of Red Hat OpenStack 13, the move to containerized overcloud services is complete.  Traditional systemd services such as RabbitMQ, Haproxy, Mariadb, etc, are all now running as containers in the overcloud.  This move to containers is meant to provide additional stability, control, and security to the platform. Future upgrades should be easier, and future deploys should be more flexible.

However, the move to containers brings with it a couple of new challenges. Operations.

The average OpenStack administrator no longer restarts services, they restart containers.

They no longer view a rabbit cluster’s status on the controller node, but rather within a container on the controller node.  Log locations have changed. Config file locations have changed.

So let’s retrain ourselves.

Logs and Config Files

So this one is not that hard. Logs for all containerized services can be found in the directory below.

# /var/log/containers/

Service configuration files are a bit harder to find, and there are several directories worth noting.

Initial configuration write occurs in the directory below.  The directory for a given service containers a full copy of configuration directories (/etc, /var/www, /root, etc).

# /var/lib/config-data/<service>

The directory shown below contains the service config files that were modified via puppet. Directories within this tree should are copied into the running container config by kolla. If you want to modify the config of a container, do it here.

#  /var/lib/config-data/puppet-generated/<service>

Paunch

Paunch is  a “utility to launch and manage containers using YAML based configuration data”.  These files can be found in the directory shown below.

/var/lib/tripleo-config/

In this directory you will find docker container startup files.  Example below.

ls | grep step_1
docker-container-startup-config-step_1.json
hashed-docker-container-startup-config-step_1.json

The “non-hashed” files are input files for their “hashed” counterparts. The “hashed” file contains a TRIPLEO_CONFIG_HASH, which is an md5 sum of the config_volumes used by each individual container.  See example below.

“TRIPLEO_CONFIG_HASH=f0cc3f9fc4b80c5aa1cb73bf4dc84100”

The TRIPLEO_CONFIG_HASH is used to notify paunch to restart a container if a config used in by the container was changed.

Using the CLI: View/Inspect Containers

The command below lists all running containers. The output will show you the container id, the container image usedcontainer health, and the container name.

docker ps

To view a list of all running or stopped containers use the command below.

docker ps --all

List docker images using the command below

docker images

Inspect a container with the command below. Among other info, the output will show you the bind mounts for the container

docker inspect

Example

docker inspect neutron_ovs_agent

The command below will show you the name of a running container. In this example we are looking for the name of the rabbit container. Here the “-f” option is used to specify a filter

docker ps -f name=rabbit

Similary, you can query all running containers in a “healthy” state. The -q option will return only the container id.

docker ps -q -f health=healthy

Using the CLI: Restarting Containers

Using the examples below you and start, stop, and restart containers.

sudo docker restart horizon
sudo docker stop horizon
sudo docker start horizon

Using the CLI: docker exec

You can use the “docker exec” command to login via bash to a container. Example below.

docker exec -it redis-bundle-docker-0 /bin/bash

You can also use the “docker exec” command to run a command in a container.  In the example below we are running docker exec to query the rabbit cluster status in the running rabbitmq container.

docker exec $(docker ps -q -f name=rabbitmq) rabbitmqctl cluster_status`

Pacemaker and Bundles

In version 1.1.17, Pacemaker introduced a new type of resource: the “bundle“.

bundle is a single resource specifying the settings, networking
requirements, and storage requirements for any number of containers
generated from the same container image.

See output below from “pcs status

Docker container set: rabbitmq-bundle [registry.access.redhat.com/rhosp13/openstack-rabbitmq:pcmklatest]
rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started pod4-overcloud-controller-0
rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started pod4-overcloud-controller-1
rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started pod4-overcloud-controller-2

The example output above shows the three containers that make up the rabbitmq bundle. One container per OpenStack controller.

Note, the exact name of the container differs from the name shown via PCS. Here the exact container name is rabbitmq-bundle-docker-0

docker ps | grep rabbitmq-bundle
0b92ca5ac66a registry.access.redhat.com/rhosp13/openstack-rabbitmq:pcmklatest “/bin/bash /usr/lo…” 46 hours ago Up 46 hours rabbitmq-bundle-docker-0

Mapping Systemd Services to Containers

The table below is taken straight from the official documentation. [1]. The left column contains the service name, the middle the list of systemd services for that service. The column on the right lists the docker container that has replaced that systemd service.

OpenStack Service Systemd Services Docker Containers
aodh openstack-aodh-evaluator

openstack-aodh-listener

openstack-aodh-notifier

httpd (openstack-aodh-api)

aodh_listener

aodh_api

aodh_notifier

aodh_evaluator

ceilometer openstack-ceilometer-central

openstack-ceilometer-collector

openstack-ceilometer-notification

httpd (openstack-ceilometer-api)

ceilometer_agent_notification

ceilometer_agent_central

cinder openstack-cinder-api

openstack-cinder-scheduler

openstack-cinder-volume

cinder_scheduler

cinder_api

openstack-cinder-volume-docker-0

glance openstack-glance-api

openstack-glance-registry

glance_api
gnocchi openstack-gnocchi-metricd

openstack-gnocchi-statsd

httpd (openstack-gnocchi-api)

gnocchi_statsd

gnocchi_api

gnocchi_metricd

heat openstack-heat-api-cfn

openstack-heat-api-cloudwatch

openstack-heat-api

openstack-heat-engine

heat_api_cfn

heat_engine

heat_api

horizon httpd (openstack-dashboard) horizon
keystone httpd (openstack-keystone) keystone
neutron neutron-dhcp-agent

neutron-l3-agent

neutron-metadata-agent

neutron-openvswitch-agent

neutron-server

neutron_ovs_agent

neutron_l3_agent

neutron_metadata_agent

neutron_dhcp

neutron_api

nova openstack-nova-api

openstack-nova-conductor

openstack-nova-consoleauth

openstack-nova-novncproxy

openstack-nova-scheduler

nova_metadata

nova_api

nova_conductor

nova_vnc_proxy

nova_consoleauth

nova_api_cron

nova_scheduler

nova_placement

panko panko_api
swift openstack-swift-account-auditor

openstack-swift-account-reaper

openstack-swift-account-replicator

openstack-swift-account

openstack-swift-container-auditor

openstack-swift-container-replicator

openstack-swift-container-updater

openstack-swift-container

openstack-swift-object-auditor

openstack-swift-object-expirer

openstack-swift-object-replicator

openstack-swift-object-updater

openstack-swift-object

openstack-swift-proxy

swift_proxy

swift_account_server

swift_container_auditor

swift_object_expirer

swift_object_updater

swift_container_replicator

swift_account_auditor

swift_object_replicator

swift_container_server

swift_rsync

swift_account_reaper

swift_account_replicator

swift_object_auditor

swift_object_server

swift_container_update

References

https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/tips_tricks.html

https://access.redhat.com/solutions/3518521

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/s1-containers-haar

 

Advertisements

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.