Linux

RHEL6: All Up in Your Face with Auditd

/ Christopher Paquin

Strongbad2kqAuditd is the userland piece of the RHEL audit tool suite. When its up and running, audit messages sent by the kenel will be send to log files that you have configured. By default, only a small and limited number of messages will be picked up by Auditd; these are mostly messages related to authentication and authorization.

Auditd has three main config files as shown below

  • /etc/sysconfig/auditd – basic configuration options
  • /etc/audit/auditd.conf – main config file
  • /etc/audit/audit.rules – auditing rules

Auditd and Syslog:

Its possible to send audit messages to a syslog. By setting active=yes in /etc/audisp/plugins.d/syslog.conf you can send all your audit messages to syslog. If your system is setup to log to a remote syslog server, then your audit messages will go along for the ride as well. Note that you can also send audit messages to a remote logging server via native audit protocol over TCP. I am not going to go into this option, but I want to make sure that we are aware that it exists.

 

Looking for Audit Events in All the Wrong Places:

Auditd includes a handy-dandy tool for searching audit logs. Ausearch. You can check out all your current audit log messages using the command below.

[root@ip-172-31-21-28 ~]# ausearch -l

Viewing audit logs in their raw format can be accomplished with the command below

[root@ip-172-31-21-28 ~]# ausearch –raw

The -a option allows you to search by audit event ids

[root@ip-172-31-21-28 ~]# ausearch -a 282

Auditd also includes ausearch, which allows you to get a quick summary of audit events, rather than trying to view massive audit logs. Usage and output shown below.

root@ip-172-31-21-28 ~]# aureport

Summary Report
======================
Range of time in logs: 07/17/2014 10:21:36.438 – 07/17/2014 19:52:49.556
Selected time for report: 07/17/2014 10:21:36 – 07/17/2014 19:52:49.556
Number of changes in configuration: 4
Number of changes to accounts, groups, or roles: 24
Number of logins: 20
Number of failed logins: 4
Number of authentications: 75
Number of failed authentications: 3
Number of users: 3
Number of terminals: 18
Number of host names: 19
Number of executables: 14
Number of files: 0
Number of AVC's: 10
Number of MAC events: 20
Number of failed syscalls: 10
Number of anomaly events: 0
Number of responses to anomaly events: 0
Number of crypto events: 244
Number of keys: 0
Number of process IDs: 203
Number of events: 1132

You can also use aureport and ausearch together. Simliar to the powerfull partnership between Batman and Robin, these two tools complement each other in ways that you can only imagine. Check out my sexy bits below.

[root@ip-172-31-21-28 ~]# ausearch –start today –raw | aureport

Summary Report
======================
Range of time in logs: 07/17/2014 10:21:36.438 – 07/17/2014 20:01:01.911
Selected time for report: 07/17/2014 10:21:36 – 07/17/2014 20:01:01.911
Number of changes in configuration: 4
Number of changes to accounts, groups, or roles: 24
Number of logins: 20
Number of failed logins: 4
Number of authentications: 75
Number of failed authentications: 3
Number of users: 3
Number of terminals: 18
Number of host names: 19
Number of executables: 14
Number of files: 0
Number of AVC's: 10
Number of MAC events: 20
Number of failed syscalls: 10
Number of anomaly events: 0
Number of responses to anomaly events: 0
Number of crypto events: 244
Number of keys: 0
Number of process IDs: 205
Number of events: 1144

Want to know another cool tool that is part of auditd? I know, its a lot to take in at one time, but I am sure that you can handle it. Using autrace you can trace and investigate system calls made by a process. 

Want to see everything that nslookup is doing? Then run the command below.

[root@ip-172-31-21-28 ~]# autrace /usr/bin/nslookup google.com

This will output a pid for you to trace with ausearch.

Trace complete. You can locate the records with 'ausearch -i -p 3359'

 

Related articles

RHEL6- Getting Up Close and Personal With Rsyslog
Linux audit files to see who made changes to a file

RHEL6: Using Advanced Log File Filtering in Rsyslog

/ Christopher Paquin

100-8591So by default when you forward logs to a syslog/rsyslog server all the logs end up in the same file (ususally configured to go to the messages file). Sometimes one may prefer to forward logs from a particular server to a separate logfile. I know for a fact that my sometimes friends in our info-sec group prefers this setup.

While managing such a setup for more than a handful of hosts would probably be a nightmare, its not actaually hard to setup on the most basic level.

First create a custom filters.conf (or whatever you want to name it) in your /etc/rsyslog.d directory. Below is the file that I created.

[root@ip-172-31-21-28 rsyslog.d]# cat filters.conf
:fromhost, isequal, "ip-172-31-25-104.ec2.internal" /var/log/ip-172-31-25-104.ec2.internal/messages
:fromhost, isequal, "ip-172-31-25-104.ec2.internal" ~

Note that you will need to include two lines in your file for each host, one with the specific file/location that you want to send your filtered log messages, and a second line that directs rsyslog to discard any messages from the specified hosts once they have been logged to the specified locations. This keeps these messages from ending up in the messages file as well as the file defined in your filter file.

 

Related articles

RHEL6 – Introduction to IPtables, Part II
HomeLab: Basic Syslog Configuration on Cisco Catalyst Devices
RHEL6- Getting Up Close and Personal With Rsyslog

RHEL6: Configuring Encrypted Remote Logging Via Rsyslog

/ Christopher Paquin

Picresized_1350512420_mummyRsyslog has the ability to forward/recieve encrypted logs using certificates. I am going to go over a very quick and dirty install and configuration. Since I am again sitting in Starbucks and do not have access to my homelab and its abundant resources, I am once again going to use Amazon EC2 instances of RHEL 6.5 for testing. The two instances that I will use we will call Server1 and Server2. Our goal here is not to build out an entire environment, but rather we are just looking to get this working on its most basic level.

Lets start with the log reciever server as we need to install a couple of packages. The first provides the certtool utility and the second provides the tls library for rsyslog.

[root@ip-172-31-21-28 ~]# yum -y install gnutls-utils.x86_64 rsyslog-gnutls

Also note that we are going to be looking at some documentation on the remote hosts to make this install and config a bit easier. You are probably going to want to install lynx so you can view the html documentation files. On my server1 instance, the rsyslog docs are in the following directory – /usr/share/doc/rsyslog-5.8.10.

This is the file that you want to take a look at. Below I am using lynx to view it.

lynx tls_cert_server.html

Now lets head on over to the sending server and install our packages again.

[root@ip-172-31-21-28 ~]# yum -y install gnutls-utils.x86_64 rsyslog-gnutls

Grab these bits from the file and modify to match your system. Then drop it in a file in /etc/rsyslog.d on your rsyslog receiving server.  I am calling my logging.conf

$ModLoad imuxsock # local messages
$ModLoad imtcp # TCP listener

# make gtls driver the default
$DefaultNetstreamDriver gtls

# certificate files
$DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem
$DefaultNetstreamDriverCertFile /rsyslog/protected/machine-cert.pem
$DefaultNetstreamDriverKeyFile /rsyslog/protected/machine-key.pem

$InputTCPServerStreamDriverAuthMode x509/name
$InputTCPServerStreamDriverPermittedPeer *.example.net
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
$InputTCPServerRun 6514 # start up listener at port 10514

Now, from the same file as you used above, grab the following giggly-bits and stick them out on the rsyslog sending server. Modify the file to match your needs and drop in /etc/rsyslog.d. To keep things simple I am going to call this file logging.conf as well

# certificate files – just CA for a client
$DefaultNetstreamDriverCAFile /path/to/contrib/gnutls/ca.pem
# set up the action
$DefaultNetstreamDriver gtls # use gtls netstream driver
$ActionSendStreamDriverMode 1 # require TLS for the connection
$ActionSendStreamDriverAuthMode anon # server is NOT authenticated
*.* @@(o)server.example.net:10514 # send (all) messages

 

Now, lets review where we are. Our config files are in place on our sending server and our recieving server. Plus we have installed all required packages. Now lets get on to the keys. This is the confusing part, so I will go through my process one step at a time. Once again you can use the built in documentation to help you get this part right.

Jump on the recieving server and check out the file below via lynx

lynx tls_cert_ca.html

Jump into your newly created keys directory.

[root@ip-172-31-21-28 ~]# cd /etc/rsyslog-keys/

And run the command below to generate the private key for your CA (Certficate Authority)

[root@ip-172-31-21-28 rsyslog-keys]# certtool –generate-privkey –outfile ca-key.pem                                        
Generating a 2048 bit RSA private key…

Now we are going to generate a self signed cert from our private key.

[root@ip-172-31-21-28 rsyslog-keys]# certtool –generate-self-signed –load-privkey ca-key.pem –outfile ca.pem

You will be asked a bunch of questions but there are only two important ones that are shown below, make sure you answer yes to both of them.

Does the certificate belong to an authority? (y/N): y
Will the certificate be used to sign other certificates? (y/N): y

Now take this key (ca.pem) and copy it onto the sending server in /etc/rsyslog-keys, or wherever you set your config files to look for them.

Now lets go back to the recieving server so we can genreate a private key.

[root@ip-172-31-21-28 rsyslog-keys]# certtool –generate-privkey –outfile key.pem –bits 2048
Generating a 2048 bit RSA private key…

Now lets generate a request key.

[root@ip-172-31-21-28 rsyslog-keys]# certtool –generate-request –load-privkey key.pem –outfile request.pem

And how lets sign the certificate request and generate a certificate.

certtool –generate-certificate –load-request request.pem –outfile cert.pem –load-ca-certificate ca.pem –load-ca-privkey ca-key.pem

This one also requires you to answer a bunch of questions. These are the two that have "yes" for the answer.

Is this a TLS web client certificate? (y/N): y
Is this also a TLS web server certificate? (y/N): y

Now at this point you should be able to start rsyslog on each server. Using the logger command on your sending server, you should be able to send a message to your local syslog and see it forward to the sender.

 

 

Related articles

RHEL6 – Configuring Apache with TLS/SSL Encryption
RHEL6- Getting Up Close and Personal With Rsyslog
Install and Configure Rsyslog Server in CentOS 6.4 / RHEL 6.4

Password Protecting Grub in RHEL 6

/ Christopher Paquin

19552780-illustration-of-cartoon-wormGrub, is the standard boot loader used by each and every Linux type operating system that I can think of. RHEL 6 uses what I guess we are now calling grub 1.o, since grub 2.0 has been released and in use by Fedora for the last few releases. You will also find that grub 2.0 has replaced grub 1.0 in RHEL 7. At some point I plan to explore grub 2 at lenght, but today is not that day (unless something strange happens before I go to bed tonight — you never know).

 

Anyway – I digress. Below is an excerpt from the current grub.conf that is in use by my EC2 RHEL 6.5 instance. I've made it really tiny to save space.

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You do not have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /, eg.
#          root (hd0)
#          kernel /boot/vmlinuz-version console=ttyS0 ro root=/dev/vda1
#          initrd /boot/initrd-[generic-]version.img
#boot=/dev/vda
default=0
timeout=1

splashimage=(hd0)/boot/grub/splash.xpm.gz

hiddenmenu
title Red Hat Enterprise Linux Server (2.6.32-431.17.1.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-431.17.1.el6.x86_64 console=ttyS0 ro root=UUID=05e7b919-2577-40a7-91fb-1ccdede87fc4 rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 xen_blkfront.sda_is_xvda=1 console=ttyS0,115200n8 console=tty0 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM rhgb quiet
        initrd /boot/initramfs-2.6.32-431.17.1.el6.x86_64.img

To configure a password to use for grub you need to use the grub-crypt command. as you can see in the example below, grub-crypt prompts you for your password, although you can also pass the password to grub-crypt in the inital command. Grub-crypt spits out your hashed and salted password.

[root@ip-172-31-22-45 ~]# grub-crypt
Password:
Retype password:
$6$V0qkcD4b/CouCY/L$.7vrHd.X6MZWzzDxvXdQn3avPwv2rUX80XKuS.Hvh1YqR11ZyINZszvJ/e0qH6VcsFMAVWAPWXtugHHYTzYj90

Now stick this password in your grub.conf using the format below.

password –encrypted $6$V0qkcD4b/CouCY/L$.7vrHd.X6MZWzzDxvXdQn3avPwv2rUX80XKuS.Hvh1YqR11ZyINZszvJ/e0qH6VcsFMAVWAPWXtugHHYTzYj90

You have a couple of options when inserting the password into the grub.conf. You can either stick this entry near the top of the file, which protects each and every boot image that you might be configured to use, or you can stick the entry directly into a particular stanza to protect a specific boot image. For example, if you are building your servers with the option to pxe-boot from grub, or wipe the disk from grub, you might want to protect these options with a password.

Also note that adding a password to grub does not keep an unauthorized user from being able to boot or reboot a server, rather what it does is protect the grub menu from being edited manually durring a reboot. This keeps nefarious users from being able to break the boot sequence and boot into single-user mode where they can compromise your system.

 

Related articles

RHEL6 – Restore Grub on MBR
Grub
RHEL6 – Mount: Could Not Find Any Free Loop Device

Fun With PAM: Working with pam_cracklib and pam_tally2

/ Christopher Paquin

Liberty-bell_13850_smPlugable Authentication Modules, or PAM, is the standard mechanism that most Unix and Linux Operatng Systems use for user credential authentication. By design, PAM is broken out into a number of files, each with a specific purpose. Before you can get started with PAM you need to understand a bit about how PAM configuration files are formatted.  So lets get into that first before we try to bite off anything more.

 

PAM Config File Standards:

PAM config files follow a standard format as shown below.

Rule Types        Control        Module [module arguments]

 

There are 4 Rule Types

  • auth – used to authenticate users/passwords
  • account – set properties for a user's account
  • password – controls password changes
  • session – sets and controls environmental variables

And there are 5 Control Types

  • required – a module that a user is required to pass
  • sufficient – a user is not required to pass a sufficient module
  • optional – these modules do not have to be passed sucessfully
  • include – these rules reference other PAM config files
  • requisite – similar to required, but if failed, no further rules are checked

 

Configuring pam_cracklib:

Pam_cracklib is used to define password complexity. It has several module arguments that can be used to define password complexity and lenght. Its most common arguments are show below

  • ucredit – when used in the following format (ucredit=-n) requires the defined number of uppercase characters in a password
  • dcredit – when used in the following format (dcredit=-n) requires the defined number of digits in a password
  • ocredit – when used in the following format (ocredit=-n) requires the defined number of other (think symbols) type charaters in a password
  • lcredit – when used in the following format (lcredit=-n) requires the defined number of lower case letters  in a password
  • minclass – defines the minimum number of different character classes that must be present in a password.
  • minlen – defines the minimum required lenght of a password.

Here's a usage example of the cracklib module from a /etc/pam.d/system-auth file. In this example try_first_pass tells pam to try to use any cached credentials, while retry allows a user to try their password 3 times before the fail this module.

password    requisite     pam_cracklib.so minlen=8 ocredit=-1 ucredit=-1 try_first_pass retry=3

 

Configuring pam_tally2:

Pam_tally2 can be used to lock users after a defined number of failed login attempts. The example below, taken from the system-auth file will lock a user after 3 failed login attempts, will automatically unlock the user after 300 seconds, and will do so quietly, without any notification to the user.

auth        required        pam_tally2.so deny=3 unlock_time=300 quiet

The command pam_tally2 can be used to list users with failed logins and can also be used to reset a user's failed login count. See reset example below

pam_tally2 –reset –u testuser

Note that pam_tally2 deprecates the faillock module.

 

Supplementary PAM Configuration Options:

Want to limit a user to a particular number of concurrent ssh sessions? You can set this up in /etc/security/limits.conf if you are calling the pam_limits.so in your pam configs. Limits.conf provides the example below. Just copy the format and you are off to the races. Remember to remove the #.

#@student        -       maxlogins       4

 

 

Related articles

How to enforce password complexity on Linux
Secure Linux Servers
You are not allowed to access to (crontab) because of pam configuration.

Quick and Dirty: Finding Duplicate UIDS and Usernames in Linux

/ Christopher Paquin

6a00d83451b31569e20167611e4ac0970b-800wiOk this one is going to be quick and dirty. I have been sitting in Starbucks for about 3 hours now in what at this point is the hardest and most uncomfortable chair in the world. At this point its almost torture.

Anyway, enough about my bottom. Ever had the need to search your /etc/password file for users with duplicate UIDS or duplicate usernames? Want to make sure that no other user, other than root, has a uid of 0? Well then today is your lucky day.

For this test I have added a couple of test users so that we have some results.

The command below is searching for any two users with the same UID, and will return a UID number of any UID that it finds twice.

[root@localhost ~]# getent passwd | cut -d: -f3 | sort -n | uniq -d
0

As you can see from the example above, the command found two users with the UID of 0. As we all know this is not good. Time to see who as been up to some funny business. Guess what, its was me. I added a user called testuser and manually changes his UID to 0.

Interesting item of note, you cannot just remove this user, as the system thinks that you are trying to remove the root user. First I need to edit /etc/passwd and manually change the user’s uid.

[root@localhost ~]# userdel testuser
userdel: user testuser is currently used by process 1
[root@localhost ~]# vi /etc/passwd
[root@localhost ~]# userdel testuser

Now lets search for any users with duplicated user names

[root@localhost ~]# getent passwd | cut -d: -f1 | sort -n | uniq -d
testuser1

Sure enough, we find that there are duplicate users with the username testuser1. A quick grep confirms this

[root@localhost ~]# grep testuser1 /etc/passwd
testuser1:x:1002:1002::/home/testuser1:/bin/bash
testuser1:x:1003:1003::/home/testuser23:/bin/bash

And again, you just cannot delete these users, you must clean them up manually.

[root@localhost ~]# userdel testuser1
Multiple entries named ‘testuser1’ in /etc/passwd. Please fix this with pwck or grpck.
userdel: cannot remove entry ‘testuser1’ from /etc/passwd

Introduction to AIDE – Advanced Intrusion Detection Environment

/ Christopher Paquin

Wally_Gator_PhotoEver heard of AIDE, neither had I. Apparently its a simple intrusion detection application that can be used to monitor file changes.  It can be confired to monitor permission, ownership, timestamp, or content changes.

Lets install it. Its in the stock Redhat repos, so its a piece of cake to install via yum.

 

[root@localhost ~]# yum -y install aide

Once installed, you can tweak the config file (/etc/aide.conf) to your liking. The stock config is pretty robust, so I am going to trim it down a bit and just monitor /etc for permission changes, and /bin for what are defined as normal changes. Normal looks at file hashes to see if the files have been modified.

/bin    NORMAL
/etc    PERMS
 

Now lets start aide

[root@localhost ~]# aide –init

AIDE, version 0.15.1

### AIDE database at /var/lib/aide/aide.db.new.gz initialized.

 

Now this part is silly, we need to rename the database created above to the name that aide is configured to use.

[root@localhost ~]# cp /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

 

Now lets check for changes.

[root@localhost ~]# aide –check

AIDE, version 0.15.1

### All files match AIDE database. Looks okay!

Hey no changes. Now lets monkey with something and see if aide catches it. In this example we are creating a new file in /etc. As seen below aide catches the new file and reports on it.

 

[root@localhost ~]# touch /etc/aide.test.change
[root@localhost ~]# aide –check
AIDE 0.15.1 found differences between database and filesystem!!
Start timestamp: 2014-07-15 19:51:14

Summary:
  Total number of files:        5054
  Added files:                  1
  Removed files:                0
  Changed files:                0

—————————————————
Added files:
—————————————————

added: /etc/aide.test.change

 

So now lets re-initialize the database, which is pretty much a snapshot.

[root@localhost ~]# aide –init

AIDE, version 0.15.1

### AIDE database at /var/lib/aide/aide.db.new.gz initialized.

 

Don't forget to overwrite the old database.

[root@localhost ~]# cp /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
cp: overwrite ‘/var/lib/aide/aide.db.gz’? yes

Now lets change the permissions on our test file and see if aide catches the change.  I'll spare you the suspense and let you know that aide did its job. See below.

 

[root@localhost ~]# chmod 777 /etc/aide.test.change
[root@localhost ~]# aide –check                    
AIDE 0.15.1 found differences between database and filesystem!!
Start timestamp: 2014-07-15 19:54:09

Summary:
  Total number of files:        5054
  Added files:                  0
  Removed files:                0
  Changed files:                2

—————————————————
Changed files:
—————————————————

changed: /etc/aide.test.change
changed: /root/.mozilla/firefox/8u03e3hs.default/sessionstore.js

—————————————————
Detailed information about changes:
—————————————————

File: /etc/aide.test.change
 Perm     : -rw-r–r–                       , -rwxrwxrwx
 ACL      : old = A:
—-
user::rw-
group::r–
other::r–
—-
                  D: <NONE>
            new = A:
—-
user::rwx
group::rwx
other::rwx
—-
                  D: <NONE>

Now aide on its own is just a simple tool, but run via cron with a bit of tuning and a bit more logic behind it and I can see it being a very useful tool. Looking forward to playing with it more.

Related articles

How To Install Aide on a DigitalOcean VPS
RHEL6 – How to Setup an Anonymous Download Only FTP Server