Mapping Virtual Networks with plotnetcfg

plot-sheet-large

Plotnetcfg is a Linux utility that you can use to scan the networking configuration on a server and output the configuration hierarchy to a file. Plotnetcfg is most useful when troubleshooting complex virtual networks with all sorts of bonds and bridges, the likes of which you will find on KVM nodes, or OpenStack Controller nodes.

You can install plot on RHEL/Centos as shown below.

# yum -y plotnetcfg.x86_64

You will also want to install the “dot” command which is installed with graphiz. See below.

# yum -y install graphviz.x86_64

Now that the bits and pieces are installed we can run the command below which outputs to PDF file named file.pd

# plotnetcfg | dot -Tpdf > file.pd

If you want to, you can also use “convert” to convert the PDF to a jpg. For example, I exported to jpg to embed below.

file

Super clean, and super easy to read and understand

 

 

RHEL 7 – Setup Software Raid Via mdadm

smarta

In this post I will walk though the steps that I used to setup software raid using mdadm on my RHEL 7.2 server.

The disks in my server are shown below.

  • /dev/sda – 64gb SSD – OS disk
  • /dev/sdb – 4TB – Unused
  • /dev/sdc – 4TB – Unused 

My plan is to create a RAID 1 mirror from the two 4TB drives (sdb and sdc) and mount the mirrored device to “/var/lib/libvirt/images” for use via KVM.

First we need to partition the disks. Note that we cannot use fdisk as does not support partition sizes over 4Tb in size.  See below.

[root@titan]# fdisk /dev/sdb

WARNING: The size of this disk is 4.0 TB (4000000000000 bytes).
DOS partition table format can not be used on drives for volumes
larger than (2199023255040 bytes) for 512-byte sectors. Use parted(1) and GUID partition table format (GPT).

 

Instead we will use parted. See steps below. Note that each step is repeated as we have two disks.

[root@titan]# parted –script /dev/sdb “mklabel gpt”
[root@titan]# parted –script /dev/sdc “mklabel gpt”
[root@titan]# parted –script /dev/sdb “mkpart primary 0% 100%”
[root@titan]# parted –script /dev/sdc “mkpart primary 0% 100%”
[root@titan]# parted –script /dev/sdb “set 1 raid on”
[root@titan]# parted –script /dev/sdc “set 1 raid on”

Now we will create the mirrored device using mdadm.

[root@titan]# mdadm –create /dev/md0 –level=raid1 –raid-devices=2 /dev/sdb1 /dev/sdc1

The new device needs to sync – we can watch its progress using mdstat.

[root@titan]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[1] sdb1[0]
3906117632 blocks super 1.2 [2/2] [UU]
[>………………..] resync = 0.0% (1893888/3906117632) finish=515.3min speed=126259K/sec
bitmap: 30/30 pages [120KB], 65536KB chunk

Note that I pretty much followed this guide line for line. Its an older article, but it checks out.

Using fdisk we can see our new mirror device.

[root@titan]# fdisk -l /dev/md0

Disk /dev/md0: 3999.9 GB, 3999864455168 bytes, 7812235264 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Let’s put a bird on it. No, let’s create a partition on it

[root@titan]# parted –script /dev/md0 “mklabel gpt”
[root@titan]# parted –script /dev/md0 “mkpart primary 0% 100%”

Great work. Now lets take a peak at our new partition.

[root@titan]# fdisk -l /dev/md0

Disk /dev/md0: 3999.9 GB, 3999864455168 bytes, 7812235264 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/md0p1 1 4294967295 2147483647+ ee GPT

Do you want to create a file system? I know I do. See below. Note the device name that we are using.

[root@titan]# mkfs -t ext4 /dev/md0p1

Now lets mount it up.

[root@titan]# mount /dev/md0p1 /var/lib/libvirt/images/

Don’t forget to add an entry to “/etc/fstab” so that our filesystem mounts at boot time.

Enable Nested Virtualization on RHEL 7

empty birds nest

Follow the steps shown below if you want to be able to run nested virtual machines on RHEL 7 via KVM.

In this particular situation I have a physical Supermicro server that I want to use to host OpenStack.

Note that my home server has Intel Xeon processors, so I first enable nested virtualization via the KVM intel module.  AMD procs use a different module.

cat << EOF > /etc/modprobe.d/kvm_intel.conf
options kvm-intel nested=1
options kvm-intel enable_shadow_vmcs=1
options kvm-intel enable_apicv=1
options kvm-intel ept=1
EOF

Also, in order to communicate with your nested VMs you will need to disable reverse path filtering, otherwise RHEL will discard any network packets in order to prevent asymmetric routing. See below.

cat << EOF > /etc/sysctl.d/98-rp-filter.conf
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
EOF

The simplest way to enable these changes is via a reboot of the physical host.

A Simple How To Guide for TMUX

brightknight-tux-hatches-3796TMUX is a terminal multiplexer, a command line tool for enabling time travel, and a utility that can be utilized for safe trans-dimensional travel. Note however, that only the first part of that last sentence is actually true.

The purpose of this post is not to teach you (or me) how to be come TMUX experts, rather we are just going to learn how to do a few simple, yet very useful things that I feel are the best features of TMUX

Creating your TMUX Config File

First you need to install tmux via your favorite package installer. Next you can drop this .tmux.conf profile in your home directory. Don’t forget to source it. This file was given to me by a fellow co-worker and seems to make tmux pretty and functional.

This .tmux.conf sets your bind-key to “ctrl-A”. From here on out we will refer to this as our bind-key


#.tmux.conf in home directory

set -g prefix C-a

#bind C-c run "tmux show-buffer | xclip -i -selection clipboard"
bind C-c run "tmux show-buffer | xclip -i -selection primary"

bind-key C-a last-window
bind-key a send-key C-a
#bind-key M-a send-prefix

bind-key h select-pane -L
bind-key ^H select-pane -L
bind-key k select-pane -U
bind-key ^K select-pane -U
bind-key j select-pane -D
bind-key ^J select-pane -D
bind-key l select-pane -R
bind-key ^L select-pane -R

bind-key J resize-pane -D 5
bind-key K resize-pane -U 5
bind-key H resize-pane -L 5
bind-key L resize-pane -R 5

bind-key S setw synchronize-panes

#set-option -g lock-command '/usr/bin/vlock'
set-option -g lock-after-time 300
bind-key X lock-client

# vi ftw
set-window-option -g mode-keys vi

# bind-key N new-session
# Enhanced new-session: set session name
bind-key N command-prompt -p name: "new-session -s %1"

#### COLOR (Solarized dark)
# default statusbar colors
set-option -g status-bg black #base02
set-option -g status-fg yellow #yellow
set-option -g status-attr default

# default window title colors
#set-window-option -g window-status-fg brightblue #base0
#set-window-option -g window-status-bg default
#set-window-option -g window-status-attr dim

# active window title colors
#set-window-option -g window-status-current-fg brightred #orange
#set-window-option -g window-status-current-bg default
#set-window-option -g window-status-current-attr bright

# pane border
#set-option -g pane-border-fg black #base02
#set-option -g pane-active-border-fg brightgreen #base01

# message text
#set-option -g message-bg black #base02
#set-option -g message-fg brightred #orange

# pane number display
#set-option -g display-panes-active-colour blue #blue
#set-option -g display-panes-colour brightred #orange

# clock
#set-window-option -g clock-mode-colour green #green

Split Window Panes

This is my favorite feature of TMUX. Need to monitor several logs at once across multiple machines without having to switch back and forth between tabs or separate terminal windows, then you are in luck. See the screenshot below for an example of this. Here I have three separate windows.


tmux

To create a virtual split run the command below.

Run the command below to split a screen vertically.

<bind-key> %

Run the command below to split a screen horizontally

<bind-key> “

Now that you have created a bunch of windows/panes you now need to move between them. Use the commands below to accomplish this.

bind-key <UP ARROW>
bind-key <DOWN ARROW>
bind-key <RIGHT ARROW>
bind-key <LEFT ARROW>

To synchronize panes run the command below. For example, lets say I jump between the three panes in my example image above and ssh to separate servers. However I then want to “tail -f /var/log/secure” on each server, but I do not want to type the command three times. Instead I sync my frames, and run the command once. The command below accomplishes this.

bind-key <S>

To unsyncronize, run the command below.

bind-key <s>

Session Management

TMUX operates very similarly to Screen when it comes to session management.

The command below will create a new session called test_session.

$ tmux new -s test_session

To detach from the test session that you just created use the command below

$ tmux detach d

Now lets re-attach to our test session.

$ tmux attach -t test_session

Ok, now after reattaching from our test session we create another session called test_session_2. See below.

$ tmux new -s test_session_2

Now lets detach from our currently active session. Same detach command as shown a few steps above

$ tmux detach d

Now lets list all our open TMUX sessions.

$ tmux list-sessions
test_session: 1 windows (created Wed May 20 13:28:34 2015) [125×33]
test_session_2: 1 windows (created Wed May 20 13:33:35 2015) [125×33]

Finally we can switch between sessions with the switch command.

$ tmux switch -t test_session

Oh, lets not forget, using the following command we can kill a session with the command below.

$ tmux kill-session -t  test_session

Also note that you can detach and close an active session with <ctl> d.

Bropages – Like Manpages but More Friendly

41KUBzxwK4L._SX300_I stumbled into this fun little tool the other day when I was introduced to it by a friend.  Bropages is a bit like man pages, but without all the hemming and hawing and information that you might not need when you are just trying to remember the proper syntax of a command.  Bropages just shows you an annotated example of how to run a command.

Before you can get started with Bropages, you need to install it. Note that I am running these commands on my Fedora21 Workstation. Your mileage may vary depending on your distro.

First you will need to install rubygems if its not already installed.

#yum -y install rubygems

Then install bropages — its a ruby gem.

#gem install bropages

Now lets take bropages out for a spin by running bro against the which command

# bro which
2 entries for which — submit your own example with “bro add which”

# locates executable
which bro

bro thanks to upvote (2)
bro …no to downvote (0)

………………………………………………………………………………………………………………………………

# Shows all install locations of an executable (in this case ruby)
which -a ruby

bro thanks 2 to upvote (2)
bro …no 2 to downvote (0)

Neat. As you can see above we have two examples of how to run the which command. Now lets try it against the chage command.

[root@fedora21 ~] $ bro chage
The chage command isn’t in our database.

* Typing bro add will let you add chage to our database!

* There’s nothing to lose by typing bro add, it will just launch an editor with instructions.

* Need help? Visit http://bropages.org/help

So bro don’t know chage, as its not in the bro database… let’s add it.

#bro add chage

We are prompted to add our short entry with a brief description

#~ Bro entry for command ‘chage’
#~ Provide a useful example for how to use ‘chage’
#~ Comments starting with #~ are removed
#~
#~ Example for command ‘man’:
#~ # Opens up the manual page for the command ‘ls’
#~ man ls
# Configure a user’s password not to expire
chage -m 0 -M 99999 -I -1 -E -1 username

There, now we have created and submitted our entry.

Redhat Satellite 5: How to Clone Security Errata to a Software Channel

space_dogFirst check to see if the errata is available to your local satellite server. To accomplish this log into your organizations satellite server and click on the “Errata” tab. Then on the left side of the page click on “Advanced Search”.

In the search box enter the RHSA number (Redhat Security Advisory Number) for the errata that you want to clone/update. In this example I am searching for RHSA-2014:1924, which is a Thunderbird security update.

If your search does not return any results, you will need to manually sync your local Satellite Server with Redhat.To accomplish this you need to ssh into your local satellite server and run the command shown below. Note that this does not update any packages/errata. This does update the list of availbile packages/errata.

/usr/bin/satellite-sync
[root@myserver ~]# satellite-sync –email
10:08:09 Red Hat Satellite – live synchronization
10:08:09 url: https://satellite.rhn.redhat.com
10:08:09 debug/output level: 1
….truncated….

Once you are able to locate the specific fix in via “Erratum Search” you may proceed to the next step. In this example, as I stated above, I am searching for RHSA-2014:1924.

clone_erratta

Now that our local Satellite server is aware of our specific errata, click on “Clone Errata” on the left side of the page. Search the page “Errata Management” for the specific fix that you want to apply. Note that the “Errata Management” page does have built in search functionality — don’t ask me why — so you must search using your browser’s own page search function.

clone_thunderbird

Once you have located the correct Security Advisory, put a check in the box and spend about 5 minutes scrolling down to the bottom of the page. Stop when your arm is tired, or once you locate the “Clone Errata” button. Obviously you want to click this.

Note that your newly added and updated errata/package may not become immediatley availible to install. You nay need to run the following commands to refresh/reload your repos.

#yum clean all

Then check for updates with the command below.

#yum check-update

Redhat 6 Minimal Kickstart Configuration with VMware Tools and Puppet Agent Install

smartaHere is my small, crude, little Kickstart configuration and post install script that I have up and running in my lab at home. Don’t expect to find anything too fancy here, as this Kickstart was purposefully built to be small and to the point. Here, the point was to spin up a VM, run through a basic install of CentOS/Redhat Linux,  and install VMware Tools along with a Puppet agent.

Note that this post assumes that you have a working Kickstart server.

First lets take a look at our kickstart file, CentOS-6.6-x86_64-minimal.ks

The section directly below kicks off our kickstart ks file. Here we set our root password (no that’s not my hash) and setup our network interface for DHCP. We do a tiny bit of disk partitioning, and setup very simple LVM. Then we choose our packages. As you can see my package list is not at all fancy, I just want to make sure that I have pretty much every package that might need for a lab VM.

# Kickstart file for RHEL 6 Minimal
# Small Disk

text
install
url --url=http://10.1.0.106/ks/loop/CentOS-6.6-x86_64-bin-DVD1
lang en_US.UTF-8
keyboard us
network --onboot yes --device eth0 --bootproto dhcp --noipv6
rootpw --iscrypted $6$X/4YYZPN$4Sv.khxXms8N8vRssR/Vl35w/m80FF5P6p7aX0D7EFfD9p734F6tU4kXdcSCoOjPiXLrVxqfKxxxxxxxxxxxq5551
firewall --disabled
authconfig --enableshadow --passalgo=sha512
selinux --permissive
timezone America/New_York

# Disk
bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb"
zerombr
clearpart --all --drives=sda
part pv.1 --grow --size=1
part /boot --fstype=ext4 --size=1024
volgroup VolGroup pv.1
logvol / --fstype=ext4 --name=lv_root --vgname=VolGroup --size=1024 --grow
logvol swap --name=lv_swap --vgname=VolGroup --size=1024

#Network
network --device=eth0 --bootproto=dhcp --nameserver=10.1.0.110

# Package Selection
%packages --nobase --excludedocs
@Base
@core
kernel-headers
wget
perl
sysstat
bind-utils
tcpdump

Now let me pause to point out the section below. This is the %pre script that I am using to prompt me for the VM hostname before the install begins. The hostname needs to be set before you install puppet on the VM, otherwise you are going to have to recreate your puppet certificates after you set properly set your hostname post install and reboot.

%pre --log=/root/ks_pre.log
#change to tty6 to get input
chvt 6
exec &lt;/dev/tty6 &gt; /dev/tty6

#Prompt for hostname
echo "What is my hostname?"
read NAME
echo "NETWORKING=yes
HOSTNAME=${NAME}" &gt; network
chvt 1

Now we run a simple post install, along with a custom post install script. It is this script that will install Vmware tools and Puppet. Myself, I prefer keeping most of my code out of the actual Kickstart ks file, however you can always jam all your code into it if you like. You will just need to validate your syntax first, as I have not tested my config this way.

%post --nochroot
# bring in hostname collected from %pre, then source it
cp network /mnt/sysimage/etc/sysconfig/network
. /mnt/sysimage/etc/sysconfig/network
# force hostname change
/mnt/sysimage/bin/hostname $HOSTNAME
#Post Install
%post --log=/root/ks-post.log
cd /root
echo "Getting the post install script - if this takes a long time check network or path"
wget http://10.1.0.106/ks/scripts/centos-6-postinstall.bash
echo "Running the post install script"
/bin/bash centos-6-postinstall.bash

Ok, so below is the post install script that I am calling in the section above. After a quick modification of my hosts file, I pull down the Puppet installer from my local Puppet server. Next we install the open source VMware tools packages, after creating the required yum repofile.

 

#!/bin/bash
#centos-6-postinstall.bash

#Switch to the 6th console and redirect all i/o
exec &lt; /dev/tty6 &gt; /dev/tty6 2&gt; /dev/tty6
chvt 6

# Lets make sure we know who the puppet server is before we get too far
echo "Adding hosts entry for puppet master"
echo "10.1.0.115 puppet puppet.lab.localdomain" &gt;&gt; /etc/hosts

## Update Via Yum - not doing this for now in order to save time
#yum -y update
# Install puppet from local puppet master
echo "Downloading and running Puppet installer"
curl -k https://10.1.0.115:8140/packages/current/install.bash | sudo bash
#Install Open Source VMware Tools
rpm --import http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-DSA-KEY.pub
rpm --import http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-RSA-KEY.pub

echo -e "[vmware-tools]\nname=VMware Tools\nbaseurl=http://packages.vmware.com/tools/esx/5.1latest/rhel6/$HOSTTYPE\nenabled=1\ngpgcheck=1" &gt; /etc/yum.repos.d/vmware-tools.repo

echo "Installing Vmware Tools"
yum -y install vmware-tools-esx-nox

#Minor grub.conf modifications
sed -i 's/rhgb quiet//' /boot/grub/grub.conf
sed -i 's/hiddenmenu//' /boot/grub/grub.conf
sed -i 's/timeout=5/timeout=10/' /boot/grub/grub.conf

#Kick off first puppet run, for some reason I think you might need to do this twice.
sleep 5
echo "Running Puppet for the first time"
puppet agent --test
puppet agent --test

#Tell us we have reached the end
echo "We have reached the end of the post-install script"

A couple of additional details to note about the post install script above. I like to modify the grub.conf so that I unhide the menu and increase the time out. I also like to make sure that we disable the Redhat graphical boot screen… I want to make sure its easy to catch any errors or miss-configurations in my kickstarts.