Enable Nested Virtualization on RHEL 7

empty birds nest

Follow the steps shown below if you want to be able to run nested virtual machines on RHEL 7 via KVM.

In this particular situation I have a physical Supermicro server that I want to use to host OpenStack.

Note that my home server has Intel Xeon processors, so I first enable nested virtualization via the KVM intel module.  AMD procs use a different module.

[code language=”css”]
cat << EOF > /etc/modprobe.d/kvm_intel.conf
options kvm-intel nested=1
options kvm-intel enable_shadow_vmcs=1
options kvm-intel enable_apicv=1
options kvm-intel ept=1
EOF
[/code]

Also, in order to communicate with your nested VMs you will need to disable reverse path filtering, otherwise RHEL will discard any network packets in order to prevent asymmetric routing. See below.

[code language=”css”]
cat << EOF > /etc/sysctl.d/98-rp-filter.conf
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
EOF
[/code]

The simplest way to enable these changes is via a reboot of the physical host.

Advertisements

A Simple How To Guide for TMUX

brightknight-tux-hatches-3796TMUX is a terminal multiplexer, a command line tool for enabling time travel, and a utility that can be utilized for safe trans-dimensional travel. Note, however, that only the first part of that last sentence is actually true.

The purpose of this post is not to teach you (or me) how to become TMUX experts, rather we are just going to learn how to do a few simple, yet very useful things that I feel are the best features of TMUX

Creating your TMUX Config File

First you need to install tmux via your favorite package installer. Next, you can drop this .tmux.conf profile in your home directory. Don’t forget to source it. This file was given to me by a fellow co-worker and seems to make tmux pretty and functional.

This .tmux.conf sets your bind-key to “ctrl-A”. From here on out we will refer to this as our bind-key

[code language=”css”]

#.tmux.conf in home directory

set -g prefix C-a

#bind C-c run "tmux show-buffer | xclip -i -selection clipboard"
bind C-c run "tmux show-buffer | xclip -i -selection primary"

bind-key C-a last-window
bind-key a send-key C-a
#bind-key M-a send-prefix

bind-key h select-pane -L
bind-key ^H select-pane -L
bind-key k select-pane -U
bind-key ^K select-pane -U
bind-key j select-pane -D
bind-key ^J select-pane -D
bind-key l select-pane -R
bind-key ^L select-pane -R

bind-key J resize-pane -D 5
bind-key K resize-pane -U 5
bind-key H resize-pane -L 5
bind-key L resize-pane -R 5

bind-key S setw synchronize-panes

#set-option -g lock-command ‘/usr/bin/vlock’
set-option -g lock-after-time 300
bind-key X lock-client

# vi ftw
set-window-option -g mode-keys vi

# bind-key N new-session
# Enhanced new-session: set session name
bind-key N command-prompt -p name: "new-session -s %1"

#### COLOR (Solarized dark)
# default statusbar colors
set-option -g status-bg black #base02
set-option -g status-fg yellow #yellow
set-option -g status-attr default

# default window title colors
#set-window-option -g window-status-fg brightblue #base0
#set-window-option -g window-status-bg default
#set-window-option -g window-status-attr dim

# active window title colors
#set-window-option -g window-status-current-fg brightred #orange
#set-window-option -g window-status-current-bg default
#set-window-option -g window-status-current-attr bright

# pane border
#set-option -g pane-border-fg black #base02
#set-option -g pane-active-border-fg brightgreen #base01

# message text
#set-option -g message-bg black #base02
#set-option -g message-fg brightred #orange

# pane number display
#set-option -g display-panes-active-colour blue #blue
#set-option -g display-panes-colour brightred #orange

# clock
#set-window-option -g clock-mode-colour green #green

[/code]

Split Window Panes

This is my favorite feature of TMUX. Need to monitor several logs at once across multiple machines without having to switch back and forth between tabs or separate terminal windows, then you are in luck. See the screenshot below for an example of this. Here I have three separate windows.


tmux

To create a virtual split run the command below.

Run the command below to split a screen vertically.

<bind-key> %

Run the command below to split a screen horizontally

<bind-key> “

Now that you have created a bunch of windows/panes you now need to move between them. Use the commands below to accomplish this.

bind-key <UP ARROW>
bind-key <DOWN ARROW>
bind-key <RIGHT ARROW>
bind-key <LEFT ARROW>

To synchronize panes, run the command below. For example, let’s say I jump between the three panes in my example image above and ssh to separate servers. However I then want to “tail -f /var/log/secure” on each server, but I do not want to type the command three times. Instead, I sync my frames and run the command once. The command below accomplishes this.

bind-key <S>

To unsyncronize, run the command below.

bind-key <S>

Session Management

TMUX operates very similarly to Screen when it comes to session management.

The command below will create a new session called test_session.

$ tmux new -s test_session

To detach from the test session that you just created use the command below

$ tmux detach d

Now let’s re-attach to our test session.

$ tmux attach -t test_session

Ok, now after reattaching from our test session we create another session called test_session_2. See below.

$ tmux new -s test_session_2

Now let’s detach from our currently active session. Same detach command as shown a few steps above

$ tmux detach d

Now let’s list all our open TMUX sessions.

$ tmux list-sessions
test_session: 1 windows (created Wed May 20 13:28:34 2015) [125×33]
test_session_2: 1 windows (created Wed May 20 13:33:35 2015) [125×33]

Finally, we can switch between sessions with the switch command.

$ tmux switch -t test_session

Oh, let’s not forget, using the following command we can kill a session with the command below.

$ tmux kill-session -t  test_session

Also, note that you can detach and close an active session with <ctl> d.

Bropages – Like Manpages but More Friendly

41KUBzxwK4L._SX300_I stumbled into this fun little tool the other day when I was introduced to it by a friend.  Bropages is a bit like man pages, but without all the hemming and hawing and information that you might not need when you are just trying to remember the proper syntax of a command.  Bropages just shows you an annotated example of how to run a command.

Before you can get started with Bropages, you need to install it. Note that I am running these commands on my Fedora21 Workstation. Your mileage may vary depending on your distro.

First you will need to install rubygems if its not already installed.

#yum -y install rubygems

Then install bropages — its a ruby gem.

#gem install bropages

Now lets take bropages out for a spin by running bro against the which command

# bro which
2 entries for which — submit your own example with “bro add which”

# locates executable
which bro

bro thanks to upvote (2)
bro …no to downvote (0)

………………………………………………………………………………………………………………………………

# Shows all install locations of an executable (in this case ruby)
which -a ruby

bro thanks 2 to upvote (2)
bro …no 2 to downvote (0)

Neat. As you can see above we have two examples of how to run the which command. Now lets try it against the chage command.

[root@fedora21 ~] $ bro chage
The chage command isn’t in our database.

* Typing bro add will let you add chage to our database!

* There’s nothing to lose by typing bro add, it will just launch an editor with instructions.

* Need help? Visit http://bropages.org/help

So bro don’t know chage, as its not in the bro database… let’s add it.

#bro add chage

We are prompted to add our short entry with a brief description

#~ Bro entry for command ‘chage’
#~ Provide a useful example for how to use ‘chage’
#~ Comments starting with #~ are removed
#~
#~ Example for command ‘man’:
#~ # Opens up the manual page for the command ‘ls’
#~ man ls
# Configure a user’s password not to expire
chage -m 0 -M 99999 -I -1 -E -1 username

There, now we have created and submitted our entry.

Redhat Satellite 5: How to Clone Security Errata to a Software Channel

space_dogFirst check to see if the errata is available to your local satellite server. To accomplish this log into your organizations satellite server and click on the “Errata” tab. Then on the left side of the page click on “Advanced Search”.

In the search box enter the RHSA number (Redhat Security Advisory Number) for the errata that you want to clone/update. In this example I am searching for RHSA-2014:1924, which is a Thunderbird security update.

If your search does not return any results, you will need to manually sync your local Satellite Server with Redhat.To accomplish this you need to ssh into your local satellite server and run the command shown below. Note that this does not update any packages/errata. This does update the list of availbile packages/errata.

/usr/bin/satellite-sync
[root@myserver ~]# satellite-sync –email
10:08:09 Red Hat Satellite – live synchronization
10:08:09 url: https://satellite.rhn.redhat.com
10:08:09 debug/output level: 1
….truncated….

Once you are able to locate the specific fix in via “Erratum Search” you may proceed to the next step. In this example, as I stated above, I am searching for RHSA-2014:1924.

clone_erratta

Now that our local Satellite server is aware of our specific errata, click on “Clone Errata” on the left side of the page. Search the page “Errata Management” for the specific fix that you want to apply. Note that the “Errata Management” page does have built in search functionality — don’t ask me why — so you must search using your browser’s own page search function.

clone_thunderbird

Once you have located the correct Security Advisory, put a check in the box and spend about 5 minutes scrolling down to the bottom of the page. Stop when your arm is tired, or once you locate the “Clone Errata” button. Obviously you want to click this.

Note that your newly added and updated errata/package may not become immediatley availible to install. You nay need to run the following commands to refresh/reload your repos.

#yum clean all

Then check for updates with the command below.

#yum check-update

Redhat 6 Minimal Kickstart Configuration with VMware Tools and Puppet Agent Install

smartaHere is my small, crude, little Kickstart configuration and post install script that I have up and running in my lab at home. Don’t expect to find anything too fancy here, as this Kickstart was purposefully built to be small and to the point. Here, the point was to spin up a VM, run through a basic install of CentOS/Redhat Linux,  and install VMware Tools along with a Puppet agent.

Note that this post assumes that you have a working Kickstart server.

First lets take a look at our kickstart file, CentOS-6.6-x86_64-minimal.ks

The section directly below kicks off our kickstart ks file. Here we set our root password (no that’s not my hash) and setup our network interface for DHCP. We do a tiny bit of disk partitioning, and setup very simple LVM. Then we choose our packages. As you can see my package list is not at all fancy, I just want to make sure that I have pretty much every package that might need for a lab VM.

[code language=”bash”]
# Kickstart file for RHEL 6 Minimal
# Small Disk

text
install
url –url=http://10.1.0.106/ks/loop/CentOS-6.6-x86_64-bin-DVD1
lang en_US.UTF-8
keyboard us
network –onboot yes –device eth0 –bootproto dhcp –noipv6
rootpw –iscrypted $6$X/4YYZPN$4Sv.khxXms8N8vRssR/Vl35w/m80FF5P6p7aX0D7EFfD9p734F6tU4kXdcSCoOjPiXLrVxqfKxxxxxxxxxxxq5551
firewall –disabled
authconfig –enableshadow –passalgo=sha512
selinux –permissive
timezone America/New_York

# Disk
bootloader –location=mbr –driveorder=sda –append="crashkernel=auto rhgb"
zerombr
clearpart –all –drives=sda
part pv.1 –grow –size=1
part /boot –fstype=ext4 –size=1024
volgroup VolGroup pv.1
logvol / –fstype=ext4 –name=lv_root –vgname=VolGroup –size=1024 –grow
logvol swap –name=lv_swap –vgname=VolGroup –size=1024

#Network
network –device=eth0 –bootproto=dhcp –nameserver=10.1.0.110

# Package Selection
%packages –nobase –excludedocs
@Base
@core
kernel-headers
wget
perl
sysstat
bind-utils
tcpdump
[/code]

Now let me pause to point out the section below. This is the %pre script that I am using to prompt me for the VM hostname before the install begins. The hostname needs to be set before you install puppet on the VM, otherwise you are going to have to recreate your puppet certificates after you set properly set your hostname post install and reboot.

[code language=”bash”]
%pre –log=/root/ks_pre.log
#change to tty6 to get input
chvt 6
exec &lt;/dev/tty6 &gt; /dev/tty6

#Prompt for hostname
echo "What is my hostname?"
read NAME
echo "NETWORKING=yes
HOSTNAME=${NAME}" &gt; network
chvt 1
[/code]

Now we run a simple post install, along with a custom post install script. It is this script that will install Vmware tools and Puppet. Myself, I prefer keeping most of my code out of the actual Kickstart ks file, however you can always jam all your code into it if you like. You will just need to validate your syntax first, as I have not tested my config this way.

[code language=”bash”]
%post –nochroot
# bring in hostname collected from %pre, then source it
cp network /mnt/sysimage/etc/sysconfig/network
. /mnt/sysimage/etc/sysconfig/network
# force hostname change
/mnt/sysimage/bin/hostname $HOSTNAME
#Post Install
%post –log=/root/ks-post.log
cd /root
echo "Getting the post install script – if this takes a long time check network or path"
wget http://10.1.0.106/ks/scripts/centos-6-postinstall.bash
echo "Running the post install script"
/bin/bash centos-6-postinstall.bash
[/code]

Ok, so below is the post install script that I am calling in the section above. After a quick modification of my hosts file, I pull down the Puppet installer from my local Puppet server. Next we install the open source VMware tools packages, after creating the required yum repofile.

 

[code language=”bash”]
#!/bin/bash
#centos-6-postinstall.bash

#Switch to the 6th console and redirect all i/o
exec &lt; /dev/tty6 &gt; /dev/tty6 2&gt; /dev/tty6
chvt 6

# Lets make sure we know who the puppet server is before we get too far
echo "Adding hosts entry for puppet master"
echo "10.1.0.115 puppet puppet.lab.localdomain" &gt;&gt; /etc/hosts

## Update Via Yum – not doing this for now in order to save time
#yum -y update
# Install puppet from local puppet master
echo "Downloading and running Puppet installer"
curl -k https://10.1.0.115:8140/packages/current/install.bash | sudo bash
#Install Open Source VMware Tools
rpm –import http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-DSA-KEY.pub
rpm –import http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-RSA-KEY.pub

echo -e "[vmware-tools]\nname=VMware Tools\nbaseurl=http://packages.vmware.com/tools/esx/5.1latest/rhel6/$HOSTTYPE\nenabled=1\ngpgcheck=1" &gt; /etc/yum.repos.d/vmware-tools.repo

echo "Installing Vmware Tools"
yum -y install vmware-tools-esx-nox

#Minor grub.conf modifications
sed -i ‘s/rhgb quiet//’ /boot/grub/grub.conf
sed -i ‘s/hiddenmenu//’ /boot/grub/grub.conf
sed -i ‘s/timeout=5/timeout=10/’ /boot/grub/grub.conf

#Kick off first puppet run, for some reason I think you might need to do this twice.
sleep 5
echo "Running Puppet for the first time"
puppet agent –test
puppet agent –test

#Tell us we have reached the end
echo "We have reached the end of the post-install script"
[/code]

A couple of additional details to note about the post install script above. I like to modify the grub.conf so that I unhide the menu and increase the time out. I also like to make sure that we disable the Redhat graphical boot screen… I want to make sure its easy to catch any errors or miss-configurations in my kickstarts.
 

RHEL6: Cool PAM Tricks – Logging Terminal Keystokes

Pam-original-cooking-spray-72450This is a neat and very useful trick that I learned today. Lets say that you want to be able to monitor and log all keystrokes that are typed as root. This is particularly useful as normally you can only log when a user uses sudo to run a command. If the user has the abilty to become root however,  then they have effectively eluded yourattempts to track their activity. Like Thomas Magnum shaking a tail, they are free to scoot around your island with the top down.

So how do you stop this from occuring? How to you log all activity and keystrokes made by root without implementing a bloated 3rd party software that will probably cost and arm and a leg? You use PAM you dingbat.
 

The secret sauce in this security burrito is the pam_tty_audit.so module. Here is how to use it,

Below is my stock /etc/pam.d/system-auth file

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        sufficient    pam_fprintd.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 500 quiet
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 500 quiet
account     required      pam_permit.so

password    requisite     pam_cracklib.so try_first_pass retry=3 type=
password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

Now look above and then look below at my modified system-auth file. Note the additonal session entry for pam_tty_audit.so.

[root@ip-172-31-21-28 pam.d]# cat system-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        sufficient    pam_fprintd.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 500 quiet
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 500 quiet
account     required      pam_permit.so

password    requisite     pam_cracklib.so try_first_pass retry=3 type=
password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_tty_audit.so enable=root
session     required      pam_limits.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

Note that you will need to add the pam_tty_audit.so call to your /etc/pam.d/password-auth as well to ensure that you capture all of root's keystrokes, no matter how they log in.

 

Related articles

Much Todo About Linux/RHEL Passwords
RHEL6 – How to Manually Logout of an Iscsi Disk
Fun With PAM: Working with pam_cracklib and pam_tally2

RHEL6: Composing Custom Auditd Rules for Fun and Profit

297247-typewriterAuditd gives you the ability to write your own custom audit rules. This functionality allows an administrator to keep a close eye on system calls, file access, and user behavior. This added functionality is especially useful in environments that are requred to adhear to compliance standards that are above and beyond normal standards. Think PCI.

Once of the simplest rules to add is a watch rule which can be set on files and directories. In the example below we are watching the /etc/passwd file for permission changes (writes and attibute changes specifically). We are creating a custom key to use for organizational purposes.

[root@ip-172-31-21-28 ~]# auditctl -w /etc/passwd -p wa -k edit_watch

Here is a cool one – lets audit all binary executions under /usr/bin.

[root@ip-172-31-21-28 ~]# auditctl -w /usr/bin -p x

Using the -l option you can list your current audit rules, and using the -s option you can see the current status of the auditd subsystem

[root@ip-172-31-21-28 ~]# auditctl -l
LIST_RULES: exit,always watch=/etc/passwd perm=wa key=edit_watch
LIST_RULES: exit,always dir=/usr/bin (0x8) perm=x

[root@ip-172-31-21-28 ~]# auditctl -s
AUDIT_STATUS: enabled=1 flag=1 pid=741 rate_limit=0 backlog_limit=320 lost=0 backlog=0

Want to get down and dirty with audit rules – you can check out the audit rules man page by running the command below.

[root@ip-172-31-21-28 ~]# man audit.rules

 

 

 

Related articles

RHEL6 – SELinux Troubleshooting II: Electric Boogaloo
Linux audit files to see who made changes to a file
A Brief Introduction to auditd