Packstack Installer Failure: “Error: Could not start Service[rabbitmq-server]: Execution of ‘/usr/bin/systemctl start rabbitmq-server’ returned 1″

openstack

Sitting in my hotel room today, I kept running into this error while trying to install OpenStack on a RHEL 7.1 VM running on my laptop. Digging through logs was not helping me one bit, and neither was trying to run “puppet apply” on the failing puppet manifests to see if I could get more info with which to troubleshoot.

Below is the specific error that I was running into. Note that my RHEL VM’s IP address is 192.168.122.75. This IP address is pre-pended to the puppet module names. Your output, will obviously, vary. Note that this output is truncated.

Applying 192.168.122.75_amqp.pp
Applying 192.168.122.75_mariadb.pp
192.168.122.75_amqp.pp: [ ERROR ]
Applying Puppet manifests [ ERROR ]

ERROR : Error appeared during Puppet run: 192.168.122.75_amqp.pp
Error: Could not start Service[rabbitmq-server]: Execution of ‘/usr/bin/systemctl start rabbitmq-server’ returned 1: Job for rabbitmq-server.service failed. See ‘systemctl status rabbitmq-server.service’ and ‘journalctl -xn’ for details.
You will find full trace in log /var/tmp/packstack/20150415-183003-mn6Kfx/manifests/192.168.122.75_amqp.pp.log
Please check log file /var/tmp/packstack/20150415-183003-mn6Kfx/openstack-setup.log for more information
Additional information:

Each and every time, the failure occurred when the installer was trying to install/start and rabbitmq-server via the puppet module amqp.pp. Attempting to start rabbitmq manually yielded the same result.

In this instance, I was trying to be fancy and I had given my VM the hostname packstack01.local (instead of sticking with localhost).

[root@packstack01 20150415-183254-Kv8u6k]# hostnamectl
Static hostname: packstack01.local
Icon name: computer
Chassis: n/a
Machine ID: ca64b7fb0c9d4459a4d313dd17b19d76
Boot ID: fc3397657ed040fca72f3d229d014b74
Virtualization: kvm
Kernel: Linux 3.10.0-229.1.2.el7.x86_64
Architecture: x86_64

Fresh out of any good ideas, I noticed that a simple nslookup on my made up hostname actually returned results. Results that I would not have expected to be valid.

[root@packstack01 20150415-183254-Kv8u6k]# nslookup packstack01.local
Server: 192.168.1.1
Address: 192.168.1.1#53

Name: packstack01.local.local
Address: 198.105.244.104
Name: packstack01.local.local
Address: 198.105.254.104

Despite never referencing my made up hostname in my answer file (by default, the answer file is generated with IP addresses only)  the Rabbitmq service was attempting to connect to itself via hostname, which obviously failed as this is a valid ip and since I was working in a hotel room without proper dns, my server was trying to connect to a machine on the opposite side of the country.

A quick bit of tinkering in the /etc/hosts file resolved this issue, and I was able to complete my install.

Note that there are probably many other reasons why one might run into this error during an OpenStack install via Packstack, however this is the one that I ran into, and thankfully it was easy to fix.

Note to self – always use localhost when working without a valid DNS entry.

An OpenStack Cloud That Frees You to Pursue Your Business

Originally posted on Red Hat Stack:

As your IT evolves toward an open, cloud-enabled data center, you can take advantage of OpenStack’s benefits: broad industry support, vendor neutrality, and fast-paced innovation.

As you move into implementation, your requirements for an OpenStack solutions shares a familiar theme: enterprise-ready, fully supported, and seamlessly-integrated products.

Can’t we just install and manage OpenStack ourselves?

OpenStack is an open source project and freely downloadable. To install and maintain OpenStack you need to recruit and retain engineers trained in Python and other technologies. If you decide to go it alone consider:

  1. How do you know OpenStack works with your hardware?
  2. Does OpenStack work with your guest instances?
  3. How do you manage and upgrade OpenStack?
  4. When you encounter problems, consider how you would solve them? Some examples:
Problem scenario Using OpenStack from Red Hat Do it yourself
Security breach Dispatch a team of experts to assess. Issue a hotfix (and contribute the fix…

View original 1,066 more words

The Red Hat Ecosystem of Integrated Cloud Products

Originally posted on Red Hat Stack:

In my prior post, I described how OpenStack from Red Hat frees  you to pursue your business with the peace of mind that your cloud is secure and stable. Red Hat has several products that enhance OpenStack to provide cloud management, virtualization, a developer platform, and scalable cloud storage.

Cloud Management with Red Hat CloudForms            

CloudForms contains three main components

  • Insight – Inventory, Reporting, Metrics Logotype_RH_Cloudforms_RGB_Black
  • Control – Eventing, Compliance, and State Management
  • Automate – Provisioning, Reconfiguration, Retirement, and Optimization

Business Benefit Use Case
One unified tool to manage virtualization and OpenStack cloud reduces the IT management overhead of multiple consoles and tools. Manage your Red Hat Virtualization, OpenStack, and VMware vSphere infrastructure with one tool, Cloud Forms.
One unified tool to manage private OpenStack and public cloud with the three components above. For temporary capacity needs, you can burst to an Amazon or OpenStack public cloud.

View original 264 more words

How to Create Non-Routable Isolated (but not Private) Vlans on a Cisco Catalyst Layer 3 Switch

data_sheet_c78-530976-1

First off let’s start out by saying that Isolated VLANs and Private VLANs are two completely different things… they are not at all the same. To a network administrator, this should make perfect sense. However, a Server or Virtualization Administrator may or may not know the different. Because of this, I hear many non-network Administrators toss around the term “Private VLAN“, when they actually mean to say “Isolated Vlan“, or more specifically what they are referring to is a “Non-Routable” VLAN.

What’s confusing is that the networks that we plan to use over our newly commissioned Non-Routable VLANs can correctly be refereed to as Private networks. They are private because no traffic can get in our out without a direct lP link to this network. However the VLANs themselves are not private, just isolated, or non-routable.

I believe that you can see where the confusion comes from.

So allow me to provide a bit of context before we go any further.

In my specific case, I need to create what are commonly (however incorrectly) referred to as Private VLANs to act as a back-end network for an OpenStack deployment. I cannot tell you how often I have heard someone make this mistake. This new VLAN, or network, needs to remain isolated from the outside world, meaning that it does not need to be able to route to any other network, or out to the internet. Rather, this new VLAN needs to send isolated traffic back and forth between network nodes deployed as part of my OpenStack Deployment. What I am describing here is not a “Private VLAN”, it is a “Non-Routable”, or “Isolated VLAN”

So please let’s make sure that we are using the correct terms.

So here is how you do it.

In my case I want to create two isolated VLANS for isolated traffic between my OpenStack nodes. Note that I am using nested virtualization, so my OpenStack nodes are themselves VMs.

First lets create what I will refer to as NR-1 (non-routable-1). We will use the VLAN id 666 as its easy to remember.

s3560#conf t
Enter configuration commands, one per line. End with CNTL/Z.
s3560(config)#vlan 666
s3560(config-vlan)#name NR-1
s3560(config-vlan)#end

Now lets create what I will refer to as NR-2. (non-routable-2)

s3560#conf t
Enter configuration commands, one per line. End with CNTL/Z.
s3560(config)#vlan 667
s3560(config-vlan)#name NR-2
s3560(config-vlan)#end

How lets check out our vlans, starting with 666

s3560#show vlan id 666

VLAN Name Status Ports
—- ——————————– ——— ——————————-
666 NR-1 active

…trunc…

Now let’s take a look at 667

s3560#show vlan id 667

VLAN Name Status Ports
—- ——————————– ——— ——————————-
667 NR-2 active

…trunc…

Note, that if I wanted to make these VLANs routable, I would need to add a layer3 interface. We are obviously not going to do that here.

Now lets add these new VLANs to our existing virtualization server trunks. We are going to do this to a range of interfaces to save time. Note that I was already allowing VLANS 101-104 and 192 on these trunks.

s3560#conf t
Enter configuration commands, one per line. End with CNTL/Z.
s3560(config)#interface range GigabitEthernet0/15 -18
s3560(config-if-range)#switchport trunk allowed vlan 101-104,192,666,667
s3560(config-if-range)#end

Now don’t forget to save our config.

s3560#copy run start
Destination filename [startup-config]?
Building configuration…
[OK]
0 bytes copied in 1.443 secs (0 bytes/sec)

OpenStack: Upload Glance Image Via the Command Line

openstack

It seems that every once and a while I run into issues uploading an image into Glance while using the WebUI. The “Create Image” function will just flat out fail without any helpful error messages.

As a work around to the failure listed above, I suggest using the simple example shown below to upload an image. Note that all commands are being run from the Controller Node.

First source your keystone credentials.

[root@rhel7 ~]# source keystonerc_admin

Then run the command below and wait for the output. If your’s looks anything like mine, then you will know that the image uploaded without issue.

NOTE: The commands below have two dashes before each option…. its hard to see this in the output.

[root@rhel7 ~(keystone_admin)]# glance image-create –name Cirros –disk-format qcow2 –container-format bare –is-public True –copy-from http://download.cirros-cloud.net/0.3.3/cirros-0.3.3-x86_64-disk.img

+——————+————————————–+
| Property | Value |
+——————+————————————–+
| checksum | None |
| container_format | bare |
| created_at | 2015-03-29T04:50:38 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | dd5937ef-c30c-465b-ba45-3f76a59164cd |
| is_public | True |
| min_disk | 0 |
| min_ram | 0 |
| name | Cirros |
| owner | 1d84fe28da3f4e0a830c52929d85c917 |
| protected | False |
| size | 13200896 |
| status | queued |
| updated_at | 2015-03-29T04:50:38 |
| virtual_size | None |
+——————+————————————–+

Feel free to verify this in the WebUI, just make sure that you refresh the page before you think you have been lulled into a false sense of accomplishment.

Alternatively, you can also upload an image directly from local disk. In my particular case I am uploading directly from a locally mounted NFS mount.

root@rhel7 storage(keystone_admin)]# glance image-create –name RHEL7-Guest –disk-format qcow2 –container-format bare –is-public True –file /storage/NAS/storage/rhel-guest-image-7.0-20140930.0.x86_64.qcow2

+——————+————————————–+
| Property | Value |
+——————+————————————–+
| checksum | 4fcc9b2b8db610187a6e7eac02608ab6 |
| container_format | bare |
| created_at | 2015-03-29T05:07:46 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | 5f6222b2-8b37-45e6-96db-c9fee8cb84b4 |
| is_public | True |
| min_disk | 0 |
| min_ram | 0 |
| name | RHEL7-Guest |
| owner | 1d84fe28da3f4e0a830c52929d85c917 |
| protected | False |
| size | 435639808 |
| status | active |
| updated_at | 2015-03-29T05:08:21 |
| virtual_size | None |
+——————+————————————–+

RHEL7: Registering a System with the Red Hat Network Using Subscription Manager

howto-draw-octopuses-tutorials_html_101880e

Note that I am about to give you the simpletons version of registering a system. Expect nothing fancy below. In  this post I will do my best to keep it brief.

If you need to register newly build RHEL 7 system using Subscription Manger rather than the ‘rhn_register’ command (which is pretty much deprecated) you will need to run the command as shown below. Including the option  “–auto-attach is pretty much the simplest method to register with RHN, as you  do not need to keep track of any of your subscription names.

[root@rhel7 ~]#  subscription-manager register –username <username> –password <password> –auto-attach

Once auto-attached you can log into rhn.redhat.com, and use the WebUI to pick and choose any additional subscriptions that you want to attach.

Once you have attached as many subscriptions as your heart desires (or you are licensed for) ,you can then run the following command to see each and every yum repository that you have access to, even if they are disabled.

[root@rhel7 ~]# yum repolist all

Depending on the number of repos that you have access to, so you might want to narrow the list down a bit, as shown below.

[root@rhel7 ~]# yum repolist all | grep -i openstack-6

rhel-7-server-openstack-6.0-debug-rpms/7Server/x86_64                           disabled
rhel-7-server-openstack-6.0-installer-debug-rpms/7Server/x86_64             disabled
rhel-7-server-openstack-6.0-installer-rpms/7Server/x86_64                        disabled
rhel-7-server-openstack-6.0-installer-source-rpms/7Server/x86_64            disabled
rhel-7-server-openstack-6.0-rpms/7Server/x86_64                                      disabled
rhel-7-server-openstack-6.0-source-rpms/7Server/x86_64                          disabled

Now you can enable the repos that you need with the command below. Note that the enable option accepts wildcards, but also note the discrepancy in the repo names in the command output above, and in the command below. If you are not passing a wildcard option on the enable command, then you will need to modify the repo names before you can run your command with any bit of success.

[root@rhel7 ~]# subscription-manager repos –enable rhel-7-server-openstack-6.0*
Repository ‘rhel-7-server-openstack-6.0-rpms’ is enabled for this system.
Repository ‘rhel-7-server-openstack-6.0-source-rpms’ is enabled for this system.
Repository ‘rhel-7-server-openstack-6.0-debug-rpms’ is enabled for this system.
Repository ‘rhel-7-server-openstack-6.0-installer-debug-rpms’ is enabled for this system.
Repository ‘rhel-7-server-openstack-6.0-installer-source-rpms’ is enabled for this system.
Repository ‘rhel-7-server-openstack-6.0-installer-rpms’ is enabled for this system.

RHEL7 – How to Set Your Hostname in Redhat Enterprise Linux 7

name-badge

Setting a server’s hostname used to be as simple as running the ‘hostname’ command and adding a “HOSTNAME” entry in /etc/sysconfig/network. However things have changed quite a bit in RHEL7. Apparently, systemd now controls setting a server’s hostname. The ‘hostname‘ command no longer works to set your hostname, however the command is still available just to confuse you.

Now in RHEL 7 you use the command ‘hostnamectl‘. Below is an example of how it works.

Here I have logged into my a RHEL 7.1 VM. You can see that the system appears to have the hostname of node1.

[root@node1 ~]# hostname
node1

However upon further inspection, I find that this is not the case. Rather, the server has a static hostname of localhost.localdomain.

[root@node1 ~]# hostnamectl
Static hostname: localhost.localdomain
Transient hostname: node1
Icon name: computer-vm
Chassis: vm
Machine ID: 4c26a2a3101947bfa2ec7d9c16824ca4
Boot ID: f58707942bd1458da48680025b6f1a53
Virtualization: vmware
CPE OS Name: cpe:/o:redhat:enterprise_linux:7.1:GA:server
Kernel: Linux 3.10.0-229.el7.x86_64
Architecture: x86_64

So lets set the hostname permanently using ‘hostnamectl’.

[root@node1 ~]# hostnamectl set-hostname node1.packy.lab.localdomain

As you can see the hostname shows correct in the output of the ‘hostname’ command

[root@node1 ~]# hostname
node1.packy.lab.localdomain

… and in the output from ‘hostnamectl’

[root@node1 ~]# hostnamectl
Static hostname: node1.packy.lab.localdomain
Icon name: computer-vm
Chassis: vm
Machine ID: 4c26a2a3101947bfa2ec7d9c16824ca4
Boot ID: f58707942bd1458da48680025b6f1a53
Virtualization: vmware
Operating System: Employee SKU
CPE OS Name: cpe:/o:redhat:enterprise_linux:7.1:GA:server
Kernel: Linux 3.10.0-229.el7.x86_64
Architecture: x86_64