This is a neat and very useful trick that I learned today. Lets say that you want to be able to monitor and log all keystrokes that are typed as root. This is particularly useful as normally you can only log when a user uses sudo to run a command. If the user has the abilty to become root however, then they have effectively eluded yourattempts to track their activity. Like Thomas Magnum shaking a tail, they are free to scoot around your island with the top down.
So how do you stop this from occuring? How to you log all activity and keystrokes made by root without implementing a bloated 3rd party software that will probably cost and arm and a leg? You use PAM you dingbat.
The secret sauce in this security burrito is the pam_tty_audit.so module. Here is how to use it,
Below is my stock /etc/pam.d/system-auth file
#%PAM-1.0 # This file is auto-generated. # User changes will be destroyed the next time authconfig is run. auth required pam_env.so auth sufficient pam_fprintd.so auth sufficient pam_unix.so nullok try_first_pass auth requisite pam_succeed_if.so uid >= 500 quiet auth required pam_deny.so
session optional pam_keyinit.so revoke session required pam_limits.so session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so
Now look above and then look below at my modified system-auth file. Note the additonal session entry for pam_tty_audit.so.
[root@ip-172-31-21-28 pam.d]# cat system-auth #%PAM-1.0 # This file is auto-generated. # User changes will be destroyed the next time authconfig is run. auth required pam_env.so auth sufficient pam_fprintd.so auth sufficient pam_unix.so nullok try_first_pass auth requisite pam_succeed_if.so uid >= 500 quiet auth required pam_deny.so
Auditd gives you the ability to write your own custom audit rules. This functionality allows an administrator to keep a close eye on system calls, file access, and user behavior. This added functionality is especially useful in environments that are requred to adhear to compliance standards that are above and beyond normal standards. Think PCI.
Once of the simplest rules to add is a watch rule which can be set on files and directories. In the example below we are watching the /etc/passwd file for permission changes (writes and attibute changes specifically). We are creating a custom key to use for organizational purposes.
[root@ip-172-31-21-28 ~]# auditctl -w /etc/passwd -p wa -k edit_watch
Here is a cool one – lets audit all binary executions under /usr/bin.
[root@ip-172-31-21-28 ~]# auditctl -w /usr/bin -p x
Using the -l option you can list your current audit rules, and using the -s option you can see the current status of the auditd subsystem
Auditd is the userland piece of the RHEL audit tool suite. When its up and running, audit messages sent by the kenel will be send to log files that you have configured. By default, only a small and limited number of messages will be picked up by Auditd; these are mostly messages related to authentication and authorization.
Its possible to send audit messages to a syslog. By setting active=yes in /etc/audisp/plugins.d/syslog.conf you can send all your audit messages to syslog. If your system is setup to log to a remote syslog server, then your audit messages will go along for the ride as well. Note that you can also send audit messages to a remote logging server via native audit protocol over TCP. I am not going to go into this option, but I want to make sure that we are aware that it exists.
Looking for Audit Events in All the Wrong Places:
Auditd includes a handy-dandy tool for searching audit logs. Ausearch. You can check out all your current audit log messages using the command below.
[root@ip-172-31-21-28 ~]# ausearch -l
Viewing audit logs in their raw format can be accomplished with the command below
[root@ip-172-31-21-28 ~]# ausearch –raw
The -a option allows you to search by audit event ids
[root@ip-172-31-21-28 ~]# ausearch -a 282
Auditd also includes ausearch, which allows you to get a quick summary of audit events, rather than trying to view massive audit logs. Usage and output shown below.
root@ip-172-31-21-28 ~]# aureport
Summary Report ====================== Range of time in logs: 07/17/2014 10:21:36.438 – 07/17/2014 19:52:49.556 Selected time for report: 07/17/2014 10:21:36 – 07/17/2014 19:52:49.556 Number of changes in configuration: 4 Number of changes to accounts, groups, or roles: 24 Number of logins: 20 Number of failed logins: 4 Number of authentications: 75 Number of failed authentications: 3 Number of users: 3 Number of terminals: 18 Number of host names: 19 Number of executables: 14 Number of files: 0 Number of AVC's: 10 Number of MAC events: 20 Number of failed syscalls: 10 Number of anomaly events: 0 Number of responses to anomaly events: 0 Number of crypto events: 244 Number of keys: 0 Number of process IDs: 203 Number of events: 1132
You can also use aureport and ausearch together. Simliar to the powerfull partnership between Batman and Robin, these two tools complement each other in ways that you can only imagine. Check out my sexy bits below.
Summary Report ====================== Range of time in logs: 07/17/2014 10:21:36.438 – 07/17/2014 20:01:01.911 Selected time for report: 07/17/2014 10:21:36 – 07/17/2014 20:01:01.911 Number of changes in configuration: 4 Number of changes to accounts, groups, or roles: 24 Number of logins: 20 Number of failed logins: 4 Number of authentications: 75 Number of failed authentications: 3 Number of users: 3 Number of terminals: 18 Number of host names: 19 Number of executables: 14 Number of files: 0 Number of AVC's: 10 Number of MAC events: 20 Number of failed syscalls: 10 Number of anomaly events: 0 Number of responses to anomaly events: 0 Number of crypto events: 244 Number of keys: 0 Number of process IDs: 205 Number of events: 1144
Want to know another cool tool that is part of auditd? I know, its a lot to take in at one time, but I am sure that you can handle it. Using autrace you can trace and investigate system calls made by a process.
Want to see everything that nslookup is doing? Then run the command below.
So by default when you forward logs to a syslog/rsyslog server all the logs end up in the same file (ususally configured to go to the messages file). Sometimes one may prefer to forward logs from a particular server to a separate logfile. I know for a fact that my sometimes friends in our info-sec group prefers this setup.
While managing such a setup for more than a handful of hosts would probably be a nightmare, its not actaually hard to setup on the most basic level.
First create a custom filters.conf (or whatever you want to name it) in your /etc/rsyslog.d directory. Below is the file that I created.
Note that you will need to include two lines in your file for each host, one with the specific file/location that you want to send your filtered log messages, and a second line that directs rsyslog to discard any messages from the specified hosts once they have been logged to the specified locations. This keeps these messages from ending up in the messages file as well as the file defined in your filter file.
Rsyslog has the ability to forward/recieve encrypted logs using certificates. I am going to go over a very quick and dirty install and configuration. Since I am again sitting in Starbucks and do not have access to my homelab and its abundant resources, I am once again going to use Amazon EC2 instances of RHEL 6.5 for testing. The two instances that I will use we will call Server1 and Server2. Our goal here is not to build out an entire environment, but rather we are just looking to get this working on its most basic level.
Lets start with the log reciever server as we need to install a couple of packages. The first provides the certtool utility and the second provides the tls library for rsyslog.
Also note that we are going to be looking at some documentation on the remote hosts to make this install and config a bit easier. You are probably going to want to install lynx so you can view the html documentation files. On my server1 instance, the rsyslog docs are in the following directory – /usr/share/doc/rsyslog-5.8.10.
This is the file that you want to take a look at. Below I am using lynx to view it.
Now lets head on over to the sending server and install our packages again.
$InputTCPServerStreamDriverAuthMode x509/name $InputTCPServerStreamDriverPermittedPeer *.example.net $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode $InputTCPServerRun 6514 # start up listener at port 10514
Now, from the same file as you used above, grab the following giggly-bits and stick them out on the rsyslog sending server. Modify the file to match your needs and drop in /etc/rsyslog.d. To keep things simple I am going to call this file logging.conf as well
# certificate files – just CA for a client $DefaultNetstreamDriverCAFile /path/to/contrib/gnutls/ca.pem # set up the action $DefaultNetstreamDriver gtls # use gtls netstream driver $ActionSendStreamDriverMode 1 # require TLS for the connection $ActionSendStreamDriverAuthMode anon # server is NOT authenticated *.* @@(o)server.example.net:10514 # send (all) messages
Now, lets review where we are. Our config files are in place on our sending server and our recieving server. Plus we have installed all required packages. Now lets get on to the keys. This is the confusing part, so I will go through my process one step at a time. Once again you can use the built in documentation to help you get this part right.
Jump on the recieving server and check out the file below via lynx
Jump into your newly created keys directory.
[root@ip-172-31-21-28 ~]# cd /etc/rsyslog-keys/
And run the command below to generate the private key for your CA (Certficate Authority)
[root@ip-172-31-21-28 rsyslog-keys]# certtool –generate-privkey –outfile ca-key.pem Generating a 2048 bit RSA private key…
Now we are going to generate a self signed cert from our private key.
This one also requires you to answer a bunch of questions. These are the two that have "yes" for the answer.
Is this a TLS web client certificate? (y/N): y Is this also a TLS web server certificate? (y/N): y
Now at this point you should be able to start rsyslog on each server. Using the logger command on your sending server, you should be able to send a message to your local syslog and see it forward to the sender.
Grub, is the standard boot loader used by each and every Linux type operating system that I can think of. RHEL 6 uses what I guess we are now calling grub 1.o, since grub 2.0 has been released and in use by Fedora for the last few releases. You will also find that grub 2.0 has replaced grub 1.0 in RHEL 7. At some point I plan to explore grub 2 at lenght, but today is not that day (unless something strange happens before I go to bed tonight — you never know).
Anyway – I digress. Below is an excerpt from the current grub.conf that is in use by my EC2 RHEL 6.5 instance. I've made it really tiny to save space.
# grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You do not have a /boot partition. This means that # all kernel and initrd paths are relative to /, eg. # root (hd0) # kernel /boot/vmlinuz-version console=ttyS0 ro root=/dev/vda1 # initrd /boot/initrd-[generic-]version.img #boot=/dev/vda default=0 timeout=1
hiddenmenu title Red Hat Enterprise Linux Server (2.6.32-431.17.1.el6.x86_64) root (hd0) kernel /boot/vmlinuz-2.6.32-431.17.1.el6.x86_64 console=ttyS0 ro root=UUID=05e7b919-2577-40a7-91fb-1ccdede87fc4 rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 xen_blkfront.sda_is_xvda=1 console=ttyS0,115200n8 console=tty0 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM rhgb quiet initrd /boot/initramfs-2.6.32-431.17.1.el6.x86_64.img
To configure a password to use for grub you need to use the grub-crypt command. as you can see in the example below, grub-crypt prompts you for your password, although you can also pass the password to grub-crypt in the inital command. Grub-crypt spits out your hashed and salted password.
You have a couple of options when inserting the password into the grub.conf. You can either stick this entry near the top of the file, which protects each and every boot image that you might be configured to use, or you can stick the entry directly into a particular stanza to protect a specific boot image. For example, if you are building your servers with the option to pxe-boot from grub, or wipe the disk from grub, you might want to protect these options with a password.
Also note that adding a password to grub does not keep an unauthorized user from being able to boot or reboot a server, rather what it does is protect the grub menu from being edited manually durring a reboot. This keeps nefarious users from being able to break the boot sequence and boot into single-user mode where they can compromise your system.