How to Remove Deactivated or Invalid Nodes from Puppet Enterprise via the CLI

January 25, 2015January 25, 2015 / Christopher Paquin / Leave a comment

Apparently just deleting a Puppet node from the Puppet Enterprise Console does not actually delete the node from the Puppet database, and free up a licenses… or at least in my case it did not. I ran into this issue tonight after removing a few of my test boxes in preparation of building a new node that I wanted to ensure was properly managed via Puppet.

As many of you know when running the free version of Puppet Enterprise you are limited to 10 managed nodes. In my Lab I am actually running only 7 nodes, however over the past few months, I have build several test VMs as part of testing my Centos 6 kickstart and Puppet bootstrap scripts. However even after deleting these nodes manually from the WebUI I noticed that I was not freeing up any licenses.

Below is the process that I had to run to manually remove nodes from my PuppetDB. First lets list all managed VMs and look for nodes that we know do not exist any more.

[root@puppet puppet]# puppet cert list -all

+ “batman.localdomain” (SHA256) 4C:76:04:38:CE:D3:4A:E0:C9:2A:C8:E6:BB:4A:92:0C:6B:39:D9:4B:7C:E4:D1:9D:0F:E2:06:FD:97:CC:72:AF
+ “robin.localdomain” (SHA256) A9:08:E8:09:B6:68:65:81:92:FC:82:93:DB:83:82:D9:A5:A2:EB:6D:DD:6E:C5:F0:45:55:A5:39:47:DB:A1:27

In the example above, I found two test VMs… Batman and Robin, which I know are no longer valid, so lets remove them and free up a couple of licenses.

[root@puppet puppet]# puppet node deactivate robin.localdomain
Submitted ‘deactivate node’ for robin.localdomain with UUID 54267132-1c23-4cf7-96f4-d6f3ff13a684

[root@puppet puppet]# puppet node deactivate batman.localdomain
Submitted ‘deactivate node’ for batman.localdomain with UUID 160d9f7c-734c-4830-8faa-23f331218b90

Once you have set the node to deactivate, lets clean out the certs.. rinse and repeat for each node to delete.

[root@puppet puppet]# puppet node clean batman.localdomain
Notice: Revoked certificate with serial 17
Notice: Removing file Puppet::SSL::Certificate batman.localdomain at ‘/etc/puppetlabs/puppet/ssl/ca/signed/batman.localdomain.pem’
batman.localdomain

Now when we return to the WebUI we should have two free licenses and can add new managed nodes without issue. Note that according to puppet themselves “that in some cases, the PE license count in the console will not decrease for up to 24 hours, but you can restart the pe-memcached service to update the license count sooner”

Getting Started With Puppet: How to Write Your First Module

December 1, 2014December 1, 2014 / Christopher Paquin / 1 Comment

This post will walk you through the process of writing your first, albeit, a very simple Puppet module. Nothing fancy here, that can come later once we get the basics down.

Before we get started let’s take a minute to go through a couple of prerequisites.

First, we will assume that you are doing your work some sort of Linux workstation or server. Second, we will assume that you have already installed and configured a Puppet Master and a Puppet Client. In this particular case, I am working on my Fedora 20 desktop. It serves as both my master and client. Last, we are assuming that have a basic site.pp and are able to run the Puppet agent without issue or error.

Now lets get down to the meat and potato. Note we are working with opensource Puppet, not Puppet Enterprise.

First lets change directory to our Puppet module directory.

#cd /etc/puppet/modules/

Now lets create a basic manifest from a template, to do this we need to generate a module. Our very simple module is going to be called harden-cron. Run the command below to generate our bare-bones module

#puppet module generate harden-cron

Now that our module has been generated lets change directories once again. This time we need to change to the manifests directory in our newly created module.

#cd harden-cron/manifests/ && vim init.pp

Now we need to edit our manifest. This is, more or less, our playbook. It defines what “work” puppet needs to do.

Our end goal is to create a manifest that will ensure that /etc/cron.allow exists, is owned by root, and has specific file permissions. We also need to make sure that /etc/cron.deny does not exist. In order to accomplish this we use the simple code shown below. Note that we have defined our class has “harden-cron”. This is the same name that we used when creating our module above.

[code language=”css”]

class harden-cron {

file {"/etc/cron.deny":
ensure => absent,
}

file {"/etc/cron.allow":
ensure => present,
owner => "root",
group => "root",
mode => "600"
}
}

[/code]

Now we need to test our syntax and make sure that we have not made any errors. Use the command below to accomplish this. No output from the script below means that you do not have any syntax errors.

#puppet parser validate init.pp

Excellent, so now we have a new module, but its not going to do anything for us until we declare it in our site.pp. So lets change directories and do a bit of editing

# cd /etc/puppet/manifests/ && vi site.pp

Continue reading →

Working with Git — How to Clone and Fork a Git Repository

December 1, 2014December 1, 2014 / Christopher Paquin / Leave a comment

So as I have stated before, I am still very new to Git. I know just enough to do a few basic tasks, but nothing much more advanced than that.

Thankfully, the process of cloning a remote Git repository is actually rather simple once you know what you are doing. Forking a repo is just as easy. Again, once you know what you are doing. Below I am going to walk you through the steps in a very simple and easy to understand way.

Cloning a Remote Repository

First lets change to a directory that we want to use as a landing zone when we download — or more accurately — clone a remote repo.

#cd /root && mkdir git

Now using the remote git URL, lets clone the fatmin github master repo @ https://github.com/Fatmin/general.git. Note that the destination of general.git name of our remote repo.

#git clone https://github.com/Fatmin/general.git

This creates the directory “general” which contains all the downloaded files, scripts, and code that we have previously checked in to our github repo. This command also initializes our pwd as a Git repo… no need to run Git init.

Forking a Remote Repository

Now lets say that we want to make a fork of the fatmin master repo. For example, lets say that we are working on a new puppet module and want to be able to test it out and then push to a new branch that we have decided to call testing.

First we need to create a new local branch. We accomplish this with the command below. As stated above, our branch is called testing.

#git checkout -b testing

Now lets grab our puppet module, script, file, code, what have you, and move it into our pwd of /root/git.

Note that even though our file is now located inside our cloned and branched repo, we need to tell Git to add it to the repo with the command below. This command will add any new files or directories in our newly created local repository.

#git add .

Now we need to commit the changes in our local repo. Here we are adding a comment along with our commit. Then we are pushing our changes back up to the remote origin (https://github.com/Fatmin/general.git) into a new branch called testing.

#git commit -m “pushing scripts to testing branch” && git push origin testing

Once we are happy with our changes, or whatever, we can then log into github and approve the merge between our master branch and our testing branch.

Getting Started With Git and Github on Fedora

November 30, 2014 / Christopher Paquin / Leave a comment

This post is going to be a very simple overview of the steps that you need to follow to get started with Git and Github on Fedora. Nothing fancy here at all, just the basics of what you need to know so that you can get up and running with pushing and pulling code. Note that these instructions assume that you have installed the Git client already, and that you are running a Linux based desktop distro with the KDE desktop environment (yes, I know that’s redundant).

Creating Your First Repository

So this is a pretty simple process. First you are going to want to navigate to https://github.com in your browser. Once there you need to go ahead and create a free account. Create a Username and Password.

Once you have completed the steps required to create a new account, and have chosen your account type (free – right?) you then need to create a New Repository. Just click on the green button on the right hand side of the page. You will need to choose a name for your new repository. Select “Initialize this repository with a README” to seed your repo so that you can perform your first clone procedure.

Since we are down to the business of keeping things simple, let’s say that you have created the user Fatmin, and that the repository that you created is called test. This would have created the repository at the following URL – –https://github.com/Fatmin/test.git.

Continue reading →

Redhat 6 Minimal Kickstart Configuration with VMware Tools and Puppet Agent Install

November 29, 2014November 29, 2014 / Christopher Paquin / Leave a comment

smarta Here is my small, crude, little Kickstart configuration and post install script that I have up and running in my lab at home. Don’t expect to find anything too fancy here, as this Kickstart was purposefully built to be small and to the point. Here, the point was to spin up a VM, run through a basic install of CentOS/Redhat Linux, and install VMware Tools along with a Puppet agent.

Note that this post assumes that you have a working Kickstart server.

First lets take a look at our kickstart file, CentOS-6.6-x86_64-minimal.ks

The section directly below kicks off our kickstart ks file. Here we set our root password (no that’s not my hash) and setup our network interface for DHCP. We do a tiny bit of disk partitioning, and setup very simple LVM. Then we choose our packages. As you can see my package list is not at all fancy, I just want to make sure that I have pretty much every package that might need for a lab VM.

[code language=”bash”]
# Kickstart file for RHEL 6 Minimal
# Small Disk

text
install
url –url=http://10.1.0.106/ks/loop/CentOS-6.6-x86_64-bin-DVD1
lang en_US.UTF-8
keyboard us
network –onboot yes –device eth0 –bootproto dhcp –noipv6
rootpw –iscrypted $6$X/4YYZPN$4Sv.khxXms8N8vRssR/Vl35w/m80FF5P6p7aX0D7EFfD9p734F6tU4kXdcSCoOjPiXLrVxqfKxxxxxxxxxxxq5551
firewall –disabled
authconfig –enableshadow –passalgo=sha512
selinux –permissive
timezone America/New_York

# Disk
bootloader –location=mbr –driveorder=sda –append="crashkernel=auto rhgb"
zerombr
clearpart –all –drives=sda
part pv.1 –grow –size=1
part /boot –fstype=ext4 –size=1024
volgroup VolGroup pv.1
logvol / –fstype=ext4 –name=lv_root –vgname=VolGroup –size=1024 –grow
logvol swap –name=lv_swap –vgname=VolGroup –size=1024

#Network
network –device=eth0 –bootproto=dhcp –nameserver=10.1.0.110

# Package Selection
%packages –nobase –excludedocs
@Base
@core
kernel-headers
wget
perl
sysstat
bind-utils
tcpdump
[/code]

Now let me pause to point out the section below. This is the %pre script that I am using to prompt me for the VM hostname before the install begins. The hostname needs to be set before you install puppet on the VM, otherwise you are going to have to recreate your puppet certificates after you set properly set your hostname post install and reboot.

[code language=”bash”]
%pre –log=/root/ks_pre.log
#change to tty6 to get input
chvt 6
exec </dev/tty6 > /dev/tty6

#Prompt for hostname
echo "What is my hostname?"
read NAME
echo "NETWORKING=yes
HOSTNAME=${NAME}" > network
chvt 1
[/code]

Now we run a simple post install, along with a custom post install script. It is this script that will install Vmware tools and Puppet. Myself, I prefer keeping most of my code out of the actual Kickstart ks file, however you can always jam all your code into it if you like. You will just need to validate your syntax first, as I have not tested my config this way.

[code language=”bash”]
%post –nochroot
# bring in hostname collected from %pre, then source it
cp network /mnt/sysimage/etc/sysconfig/network
. /mnt/sysimage/etc/sysconfig/network
# force hostname change
/mnt/sysimage/bin/hostname $HOSTNAME
#Post Install
%post –log=/root/ks-post.log
cd /root
echo "Getting the post install script – if this takes a long time check network or path"
wget http://10.1.0.106/ks/scripts/centos-6-postinstall.bash
echo "Running the post install script"
/bin/bash centos-6-postinstall.bash
[/code]

Ok, so below is the post install script that I am calling in the section above. After a quick modification of my hosts file, I pull down the Puppet installer from my local Puppet server. Next we install the open source VMware tools packages, after creating the required yum repofile.

[code language=”bash”]
#!/bin/bash
#centos-6-postinstall.bash

#Switch to the 6th console and redirect all i/o
exec < /dev/tty6 > /dev/tty6 2> /dev/tty6
chvt 6

# Lets make sure we know who the puppet server is before we get too far
echo "Adding hosts entry for puppet master"
echo "10.1.0.115 puppet puppet.lab.localdomain" >> /etc/hosts

## Update Via Yum – not doing this for now in order to save time
#yum -y update
# Install puppet from local puppet master
echo "Downloading and running Puppet installer"
curl -k https://10.1.0.115:8140/packages/current/install.bash | sudo bash
#Install Open Source VMware Tools
rpm –import http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-DSA-KEY.pub
rpm –import http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-RSA-KEY.pub

echo -e "[vmware-tools]\nname=VMware Tools\nbaseurl=http://packages.vmware.com/tools/esx/5.1latest/rhel6/$HOSTTYPE\nenabled=1\ngpgcheck=1" > /etc/yum.repos.d/vmware-tools.repo

echo "Installing Vmware Tools"
yum -y install vmware-tools-esx-nox

#Minor grub.conf modifications
sed -i ‘s/rhgb quiet//’ /boot/grub/grub.conf
sed -i ‘s/hiddenmenu//’ /boot/grub/grub.conf
sed -i ‘s/timeout=5/timeout=10/’ /boot/grub/grub.conf

#Kick off first puppet run, for some reason I think you might need to do this twice.
sleep 5
echo "Running Puppet for the first time"
puppet agent –test
puppet agent –test

#Tell us we have reached the end
echo "We have reached the end of the post-install script"
[/code]

A couple of additional details to note about the post install script above. I like to modify the grub.conf so that I unhide the menu and increase the time out. I also like to make sure that we disable the Redhat graphical boot screen… I want to make sure its easy to catch any errors or miss-configurations in my kickstarts.

Puppet: How Not To Generate a Certificate with Your Correct Hostname

November 9, 2014December 1, 2014 / Christopher Paquin

So, I’ve been hacking away in my homelab as of late, building out a CentOS kickstart server, a Git server, and a puppet server. Right now, I am working on how to roll my puppet agent installs into my kickstart process. I just started on this, so I have yet to nail it down.

So currently, when kicking a VM, I am not yet setting my new CentOS node’s hostname before the install process. Sadly I am setting it manually as I am still building my kickstarts, and they are no where near where I want them to be.

Well, this whole hostname mumbo-jumbo just creates all sorts of issues for puppet… the hostname is one thing initially, then puppet installs as part of the post, and the hostname is set manually to finalize the install. Well this is no good, as you are are not going to be able to add your new node properly until you step in and provide a bit of manual persuasion.

Now while its not hard to find documentation on how to troubleshoot puppet node and master certificate issues — see here and here for example — none of it was written to help troubleshoot the mess that I had created.

Here was my specfic error.

Error: Could not request certificate: The certificate retrieved from the master does not match the agent’s private key.
Certificate fingerprint: BE:B6:B6:5E:AC:B8: ..truncated

And here verbatim, is the output that you get in response to the error above.

To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.

On the master:
puppet cert clean localhost.localdomain

On the agent:
rm -f /etc/puppetlabs/puppet/ssl/certs/localhost.localdomain.pem
puppet agent -t

So we try that and it doesn’t work. The next cert I generate identifies my node as localhost again.

So heres how to fix the issue.

# rm -rf /etc/puppetlabs/puppet/ssl

Now before we generate another certificate for our node, lets test what hostname a new cert would have using the command below.

#puppet agent –verbose –configprint certname

If the command above does not spit out the correct hostname, then you my friend, are in luck. Edit the file below

# vi /etc/puppetlabs/puppet/puppet.conf

Now change the entry below by removing the localhost.localdomain, and replacing that mess with the correct hostname

certname = correcthostname.localdomain

Now kickoff a puppet run on the node

#puppet agent -t

Log into the UI, or ssh into the puppet master, and accept the new node request.

Kick off another puppet run after you have accepted the request to seal the deal and update the new node properly.

How to Create a Vagrant Base Box from an Existing One

Some brief notes on Docker

Getting Started with Git: Creating a Git Repo

October 31, 2014November 28, 2014 / Christopher Paquin

So first off let me start by saying that I know that there is a ton of information out there on how to get started with Git. Heck, when you create your repo in GitLab it spits these instructions right out in front of your nose. However, what I have found is that most instructions tell you what to do to get started with git, however they do not tell you exactly what you are doing. You end up running a few command and then sit back and try to figure out what you actually just did.

That being said, getting started with Git has been the hardest part of the process, as most of us traditional grey-beard sysadmins are not that familiar with code management. When I was first getting started in technology, there were developers and there were sysadmins, and those two worlds were extremely separate. Now, we as we enter the age of DEVOPS and automation these two worlds are once again colliding (like they did in the beginning, but more on that another day).

In my lab I decided to build a stand alone vm for Git and Puppet. After doing some research – and asking others– on how and what to do to get Git up and running with a nice front end web interface, I decided to get started by installing the GitLab Omnibus package for Centos 6. This process was quick, easy, and painless.

Once I had a working webUI up and running it was time to create my first repo. Instead of trying to accomplish this from my Fedora workstaion, I just clicked on the "New Project" button on the GitLab dashboard and created an empty repo called "General_Scripts".

Back on my Fedora workstation I created a new directory in my home dir called git, and inside that directory I created a directory called "General_Scripts" as I had done in the webUI.

Now it was time to use Git.

First off you need to configure a few global Git options. This only needs to be done once.

#git config –global user.name "Fatmin"

#git config –global user.email "fatmin@fatmin.com"

Once these global configs are set you can then you can move on to seeding your repo. Here you see my change directories to the Global_Scripts directory and tell Git to initialize this directory as a repo.

#cd Global_Scripts

#git init

Now lets create a file and add it to the repo. Here were are going to create a simple README containing a description of my new repo. The instructions do not tell you that you have to put anything in this initial file, however what good is an empty README anyway

#vi README.md

Now tell git that this file needs to be added to the "General_Scripts" repo that we created a few steps ago.

#git add README.md

Now lets commit that file with a nice little comment. Commit comments should describe what the file we added or what we changed in an existing file

#git commit -m "Added README.md"

Now we need to tell our local git command what remote repo we are going to sync to. Note my git repo url is puppet.lab.localdomain, fatmin is my user's namespace, and General_Scripts is my repo.