How to disable Cloud-Init in a RHEL Cloud Image

happy_cloud1600

So this one is pretty simple. However, I found a lot of misinformation along the way, so I figured that I would jot the proper (and most simple) process here.

Symptoms: a RHEL (or variant) VM that takes a very long time to boot. On the VM console, you can see the following output while the VM boot process is stalled and waiting for a timeout. Note that the message below has nothing to do with cloud init, but its the output that I have most often seen on the console while waiting for a VM to boot.

[106.325574} random: crng init done

Note that I have run into this issue in both OpenStack (when booting from external provider networks) and in KVM.

Upon initial boot of the VM, run the command below.

touch /etc/cloud/cloud-init.disabled

Seriously, that’s it. No need to disable or remove cloud-init services. See reference.

 

 

Linux: Using Tcpdump to Capture LLDP Info

inner-banner-itnetworkaudit

According to Wikipedia, “Link Layer Discovery Protocol (LLDP) is a vendor-neutral link layer protocol in the Internet Protocol Suite used by network devices for advertising their identity, capabilities, and neighbors on an IEEE 802 local area network, principally wired Ethernet”

LLDP is often what you will find running on non-Cisco switches and routers (which usually run CDP). If you want to use tcpdump to capture northbound switch port information, you can use the example below as a guide.

 

# tcpdump -nn -v -i p1p2 ether proto 0x88cc
tcpdump: WARNING: p1p2: no IPv4 address assigned
tcpdump: listening on p1p2, link-type EN10MB (Ethernet), capture size 65535 bytes
19:00:12.559556 LLDP, length 218
Chassis ID TLV (1), length 7
Subtype MAC address (4): f4:8e:38:28:b6:87
Port ID TLV (2), length 11
Subtype Interface Name (5): ethernet11
Time to Live TLV (3), length 2: TTL 120s
Port Description TLV (4), length 39: Big Cloud Fabric Switch Port ethernet11
System Name TLV (5), length 22: SW01
..trunc..

 

 

 

Mapping Virtual Networks with plotnetcfg

plot-sheet-large

Plotnetcfg is a Linux utility that you can use to scan the networking configuration on a server and output the configuration hierarchy to a file. Plotnetcfg is most useful when troubleshooting complex virtual networks with all sorts of bonds and bridges, the likes of which you will find on KVM nodes, or OpenStack Controller nodes.

You can install plot on RHEL/Centos as shown below.

# yum -y plotnetcfg.x86_64

You will also want to install the “dot” command which is installed with graphiz. See below.

# yum -y install graphviz.x86_64

Now that the bits and pieces are installed we can run the command below which outputs to PDF file named file.pd

# plotnetcfg | dot -Tpdf > file.pd

If you want to, you can also use “convert” to convert the PDF to a jpg. For example, I exported to jpg to embed below.

file

Super clean, and super easy to read and understand

 

 

RHEL 7 – Setup Software Raid Via mdadm

smarta

In this post I will walk though the steps that I used to setup software raid using mdadm on my RHEL 7.2 server.

The disks in my server are shown below.

  • /dev/sda – 64gb SSD – OS disk
  • /dev/sdb – 4TB – Unused
  • /dev/sdc – 4TB – Unused 

My plan is to create a RAID 1 mirror from the two 4TB drives (sdb and sdc) and mount the mirrored device to “/var/lib/libvirt/images” for use via KVM.

First we need to partition the disks. Note that we cannot use fdisk as does not support partition sizes over 4Tb in size.  See below.

[root@titan]# fdisk /dev/sdb

WARNING: The size of this disk is 4.0 TB (4000000000000 bytes).
DOS partition table format can not be used on drives for volumes
larger than (2199023255040 bytes) for 512-byte sectors. Use parted(1) and GUID partition table format (GPT).

 

Instead we will use parted. See steps below. Note that each step is repeated as we have two disks.

[root@titan]# parted –script /dev/sdb “mklabel gpt”
[root@titan]# parted –script /dev/sdc “mklabel gpt”
[root@titan]# parted –script /dev/sdb “mkpart primary 0% 100%”
[root@titan]# parted –script /dev/sdc “mkpart primary 0% 100%”
[root@titan]# parted –script /dev/sdb “set 1 raid on”
[root@titan]# parted –script /dev/sdc “set 1 raid on”

Now we will create the mirrored device using mdadm.

[root@titan]# mdadm –create /dev/md0 –level=raid1 –raid-devices=2 /dev/sdb1 /dev/sdc1

The new device needs to sync – we can watch its progress using mdstat.

[root@titan]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[1] sdb1[0]
3906117632 blocks super 1.2 [2/2] [UU]
[>………………..] resync = 0.0% (1893888/3906117632) finish=515.3min speed=126259K/sec
bitmap: 30/30 pages [120KB], 65536KB chunk

Note that I pretty much followed this guide line for line. Its an older article, but it checks out.

Using fdisk we can see our new mirror device.

[root@titan]# fdisk -l /dev/md0

Disk /dev/md0: 3999.9 GB, 3999864455168 bytes, 7812235264 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Let’s put a bird on it. No, let’s create a partition on it

[root@titan]# parted –script /dev/md0 “mklabel gpt”
[root@titan]# parted –script /dev/md0 “mkpart primary 0% 100%”

Great work. Now lets take a peak at our new partition.

[root@titan]# fdisk -l /dev/md0

Disk /dev/md0: 3999.9 GB, 3999864455168 bytes, 7812235264 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/md0p1 1 4294967295 2147483647+ ee GPT

Do you want to create a file system? I know I do. See below. Note the device name that we are using.

[root@titan]# mkfs -t ext4 /dev/md0p1

Now lets mount it up.

[root@titan]# mount /dev/md0p1 /var/lib/libvirt/images/

Don’t forget to add an entry to “/etc/fstab” so that our filesystem mounts at boot time.

Enable Nested Virtualization on RHEL 7

empty birds nest

Follow the steps shown below if you want to be able to run nested virtual machines on RHEL 7 via KVM.

In this particular situation I have a physical Supermicro server that I want to use to host OpenStack.

Note that my home server has Intel Xeon processors, so I first enable nested virtualization via the KVM intel module.  AMD procs use a different module.

cat << EOF > /etc/modprobe.d/kvm_intel.conf
options kvm-intel nested=1
options kvm-intel enable_shadow_vmcs=1
options kvm-intel enable_apicv=1
options kvm-intel ept=1
EOF

Also, in order to communicate with your nested VMs you will need to disable reverse path filtering, otherwise RHEL will discard any network packets in order to prevent asymmetric routing. See below.

cat << EOF > /etc/sysctl.d/98-rp-filter.conf
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
EOF

The simplest way to enable these changes is via a reboot of the physical host.

A Simple How To Guide for TMUX

brightknight-tux-hatches-3796TMUX is a terminal multiplexer, a command line tool for enabling time travel, and a utility that can be utilized for safe trans-dimensional travel. Note, however, that only the first part of that last sentence is actually true.

The purpose of this post is not to teach you (or me) how to become TMUX experts, rather we are just going to learn how to do a few simple, yet very useful things that I feel are the best features of TMUX

Creating your TMUX Config File

First you need to install tmux via your favorite package installer. Next, you can drop this .tmux.conf profile in your home directory. Don’t forget to source it. This file was given to me by a fellow co-worker and seems to make tmux pretty and functional.

This .tmux.conf sets your bind-key to “ctrl-A”. From here on out we will refer to this as our bind-key


#.tmux.conf in home directory

set -g prefix C-a

#bind C-c run "tmux show-buffer | xclip -i -selection clipboard"
bind C-c run "tmux show-buffer | xclip -i -selection primary"

bind-key C-a last-window
bind-key a send-key C-a
#bind-key M-a send-prefix

bind-key h select-pane -L
bind-key ^H select-pane -L
bind-key k select-pane -U
bind-key ^K select-pane -U
bind-key j select-pane -D
bind-key ^J select-pane -D
bind-key l select-pane -R
bind-key ^L select-pane -R

bind-key J resize-pane -D 5
bind-key K resize-pane -U 5
bind-key H resize-pane -L 5
bind-key L resize-pane -R 5

bind-key S setw synchronize-panes

#set-option -g lock-command '/usr/bin/vlock'
set-option -g lock-after-time 300
bind-key X lock-client

# vi ftw
set-window-option -g mode-keys vi

# bind-key N new-session
# Enhanced new-session: set session name
bind-key N command-prompt -p name: "new-session -s %1"

#### COLOR (Solarized dark)
# default statusbar colors
set-option -g status-bg black #base02
set-option -g status-fg yellow #yellow
set-option -g status-attr default

# default window title colors
#set-window-option -g window-status-fg brightblue #base0
#set-window-option -g window-status-bg default
#set-window-option -g window-status-attr dim

# active window title colors
#set-window-option -g window-status-current-fg brightred #orange
#set-window-option -g window-status-current-bg default
#set-window-option -g window-status-current-attr bright

# pane border
#set-option -g pane-border-fg black #base02
#set-option -g pane-active-border-fg brightgreen #base01

# message text
#set-option -g message-bg black #base02
#set-option -g message-fg brightred #orange

# pane number display
#set-option -g display-panes-active-colour blue #blue
#set-option -g display-panes-colour brightred #orange

# clock
#set-window-option -g clock-mode-colour green #green

Split Window Panes

This is my favorite feature of TMUX. Need to monitor several logs at once across multiple machines without having to switch back and forth between tabs or separate terminal windows, then you are in luck. See the screenshot below for an example of this. Here I have three separate windows.


tmux

To create a virtual split run the command below.

Run the command below to split a screen vertically.

<bind-key> %

Run the command below to split a screen horizontally

<bind-key> “

Now that you have created a bunch of windows/panes you now need to move between them. Use the commands below to accomplish this.

bind-key <UP ARROW>
bind-key <DOWN ARROW>
bind-key <RIGHT ARROW>
bind-key <LEFT ARROW>

To synchronize panes, run the command below. For example, let’s say I jump between the three panes in my example image above and ssh to separate servers. However I then want to “tail -f /var/log/secure” on each server, but I do not want to type the command three times. Instead, I sync my frames and run the command once. The command below accomplishes this.

bind-key <S>

To unsyncronize, run the command below.

bind-key <S>

Session Management

TMUX operates very similarly to Screen when it comes to session management.

The command below will create a new session called test_session.

$ tmux new -s test_session

To detach from the test session that you just created use the command below

$ tmux detach d

Now let’s re-attach to our test session.

$ tmux attach -t test_session

Ok, now after reattaching from our test session we create another session called test_session_2. See below.

$ tmux new -s test_session_2

Now let’s detach from our currently active session. Same detach command as shown a few steps above

$ tmux detach d

Now let’s list all our open TMUX sessions.

$ tmux list-sessions
test_session: 1 windows (created Wed May 20 13:28:34 2015) [125×33]
test_session_2: 1 windows (created Wed May 20 13:33:35 2015) [125×33]

Finally, we can switch between sessions with the switch command.

$ tmux switch -t test_session

Oh, let’s not forget, using the following command we can kill a session with the command below.

$ tmux kill-session -t  test_session

Also, note that you can detach and close an active session with <ctl> d.

Bropages – Like Manpages but More Friendly

41KUBzxwK4L._SX300_I stumbled into this fun little tool the other day when I was introduced to it by a friend.  Bropages is a bit like man pages, but without all the hemming and hawing and information that you might not need when you are just trying to remember the proper syntax of a command.  Bropages just shows you an annotated example of how to run a command.

Before you can get started with Bropages, you need to install it. Note that I am running these commands on my Fedora21 Workstation. Your mileage may vary depending on your distro.

First you will need to install rubygems if its not already installed.

#yum -y install rubygems

Then install bropages — its a ruby gem.

#gem install bropages

Now lets take bropages out for a spin by running bro against the which command

# bro which
2 entries for which — submit your own example with “bro add which”

# locates executable
which bro

bro thanks to upvote (2)
bro …no to downvote (0)

………………………………………………………………………………………………………………………………

# Shows all install locations of an executable (in this case ruby)
which -a ruby

bro thanks 2 to upvote (2)
bro …no 2 to downvote (0)

Neat. As you can see above we have two examples of how to run the which command. Now lets try it against the chage command.

[root@fedora21 ~] $ bro chage
The chage command isn’t in our database.

* Typing bro add will let you add chage to our database!

* There’s nothing to lose by typing bro add, it will just launch an editor with instructions.

* Need help? Visit http://bropages.org/help

So bro don’t know chage, as its not in the bro database… let’s add it.

#bro add chage

We are prompted to add our short entry with a brief description

#~ Bro entry for command ‘chage’
#~ Provide a useful example for how to use ‘chage’
#~ Comments starting with #~ are removed
#~
#~ Example for command ‘man’:
#~ # Opens up the manual page for the command ‘ls’
#~ man ls
# Configure a user’s password not to expire
chage -m 0 -M 99999 -I -1 -E -1 username

There, now we have created and submitted our entry.