You can run into this error when attempting to join a node into a Rabbitmq cluster when the cluster believes that a particular node is already a member of the cluster. I have run into this issue a few times and is usually seen when attempting to recover from a crash of an OpenStack controller.
I have run into this issue a few times and is usually seen when attempting to recover from a crash of an OpenStack controller.
Below are the steps to resolve the issue.
The error below is seen when attempting to add a node back into the cluster.
INFO REPORT==== 27-Jan-2017::16:57:22 ===
Already member of cluster: [rabbit@nodectrl2,rabbit@nodectrl1,
rabbit@nodectrl0]
We check the cluster status for confirmation.
root@nodectrl1 rabbitmq]# rabbitmqctl cluster_status
Cluster status of node rabbit@nodectrl1 …
[{nodes,[{disc,[rabbit@nodectrl0,rabbit@nodectrl1,
rabbit@nodectrl2]}]},
{running_nodes,[rabbit@nodectrl2,rabbit@nodectrl1]},
{cluster_name,<<“rabbit@nodectrl0.localdomain”>>},
{partitions,[]},
{alarms,[{rabbit@nodectrl2,[]},{rabbit@nodectrl1,[]}]}]
Now we force the cluster to forget the affected node.
[root@nodectrl1 rabbitmq]# rabbitmqctl forget_cluster_node rabbit@nodectrl0
Removing node rabbit@nodectrl0 from cluster …
We then check the cluster status to ensure that it has been removed from the cluster.
[root@nodectrl1 rabbitmq]# rabbitmqctl cluster_status
Cluster status of node rabbit@nodectrl1 …
[{nodes,[{disc,[rabbit@nodectrl1,rabbit@nodectrl2]}]},
{running_nodes,[rabbit@nodectrl2,rabbit@nodectrl1]},
{cluster_name,<<“rabbit@nodectrl0.localdomain”>>},
{partitions,[]},
{alarms,[{rabbit@nodectrl2,[]},{rabbit@nodectrl1,[]}]}]
We can now add our node back into the cluster.
[root@nodectrl1 rabbitmq]# rabbitmqctl -n nodectrl1 join_cluster rabbit@nodectrl0.localdomain