You can run into this error when attempting to join a node into a Rabbitmq cluster when the cluster believes that a particular node is already a member of the cluster. I have run into this issue a few times and is usually seen when attempting to recover from a crash of an OpenStack controller.
I have run into this issue a few times and is usually seen when attempting to recover from a crash of an OpenStack controller.
Below are the steps to resolve the issue.
The error below is seen when attempting to add a node back into the cluster.
INFO REPORT==== 27-Jan-2017::16:57:22 ===
Already member of cluster: [rabbit@nodectrl2,rabbit@nodectrl1,
We check the cluster status for confirmation.
root@nodectrl1 rabbitmq]# rabbitmqctl cluster_status
Cluster status of node rabbit@nodectrl1 …
Now we force the cluster to forget the affected node.
In this lab, we are going to deploy a functional, yet simple, Overcloud via the Tripleo WebUI using Virtual Machines. Our test deployment will consist of three Overcloud Controller Nodes (configured with HA) and one Overcloud Compute Node.
Undercloud VM Details
Note that we have installed squid on our hypervisor node and are using that node to proxy all web traffic to the undercloud controller. This also requires configuring your web browser to use the hypervisor as its proxy.
This article documents the configuration used to configure SR-IOV in OSP 8/Liberty on Dell hardware
Compute Node Configuration
This section will outline the changes needed to configure SR-IOV on each Compute Node.
Bios Configuration on Dell Compute Nodes
First, you will need to ssh to the drac of each Compute Node. Then, type the command below to enter racadm command line.
Type the command below to enable SRIOV.
#racadm set BIOS.IntegratedDevices.SriovGlobalEnable Enabled
RAC1017: Successfully modified the object value and the change is in
To apply modified value, create a configuration job and reboot
the system. To create the commit and reboot jobs, use “jobqueue”
command. For more information about the “jobqueue” command, see RACADM
Type the command below to verify your configuration.
This tells nova that all VFs belonging to the physical interface, “p1p1“, are allowed to be passed through to VMs and belong to the neutron provider network “sriov_net1” and all VFs belonging to the physical interface, “p3p1“, are allowed to be passed through for the network “sriov_net2“.
Restart nova-compute on each compute node with the command shown below.
#systemctl restart openstack-nova-compute
Install and Enable Neutron Sriov-Agent (Compute)
Note that the sriov-agent is not required, however, we are going to install and configure it anyway.
Plotnetcfg is a Linux utility that you can use to scan the networking configuration on a server and output the configuration hierarchy to a file. Plotnetcfg is most useful when troubleshooting complex virtual networks with all sorts of bonds and bridges, the likes of which you will find on KVM nodes, or OpenStack Controller nodes.
You can install plot on RHEL/Centos as shown below.
# yum -y plotnetcfg.x86_64
You will also want to install the “dot” command which is installed with graphiz. See below.
# yum -y install graphviz.x86_64
Now that the bits and pieces are installed we can run the command below which outputs to PDF file named file.pd
# plotnetcfg | dot -Tpdf > file.pd
If you want to, you can also use “convert” to convert the PDF to a jpg. For example, I exported to jpg to embed below.
Super clean, and super easy to read and understand