Small linux deployments

I always find our traffic patterns to be extremely interesting, from both an administrative and also a human behavior perspective. As could be expected, our traffic was way down for the holiday weekend.

This is an example traffic graph for one of our servers.

It’s even more pronounced if you drill it down to only smtp traffic.

Kudos to Tobi Oetiker for writing mrtg, smokeping, and rrdtool, one of which almost every admin uses, and some of us use all of them. If you use his stuff, consider getting him something off of his (in)famous wish-list

As originally reported by Artem Nosulchik The NSA recently released their security configuration guide for Redhat Enterprise 5. This is an excellent (and surprisingly practical) jumping off point for securely configuring a server.

It is written for the mid-level admin, so it expects knowledge of how to get around Linux but doesn’t expect you to be an selinux guru. It could prove especially helpful where you need a reference-able set of best practices you followed for a server that will be audited (i.e. bank web server or server that collects HIPAA-protected data) without spending a fortune on consulting. It certainly leaves a few things out (like reboots after kernel updates), and it also recommends a few things that makes life difficult or impossible to the mission, but in a whole it’s the best concise set of best practices I’ve found. Here’s a quick overview of some things it covers (by no means all inclusive):

• Bootloader/bios configuration: password protect the bootloader to prevent single user mode, shut off external media booting, and password protect the bios setup
• Set up automatic yum updates. As seen in previous posts I tend to disagree with this, but it’s understandable in a high security environment
• Install and configure aide, a tripwire-like IDS. Write a baseline copy of the database to CD and store securely. I tend to think that the latter version of this is of limited usefulness because so many packages get updated over time
• Configure fstab: Add nodev to all except root, add nodev/nosuid/noexec to removable media partitions
• Restrict console permissions to root user, so a nonroot user at the console doesn’t get access to the floppy drive, etc
• Restrict or remove USB support. Watch out to not kill the USB mouse/keyboard (I wonder if they made this mistake in testing.)
• Disable automounting
• Verify permissions of /etc/passwd, /etc/shadow, /etc/group
• Find list of setuid programs and tailor to application (They include a list of the ones that are on by default, and when they recommend turning each off)
• Enable exec-shield (kernel extensions that prevent buffer overflows)
• Restrict access to su to only administrative users. Restrict root-logins to the console.
• Remove telnet, rlogin, rsh, nis, and rcp. Strongly consider removing xinetd

They also have a bunch of service specific best practices that you definitely should have a look at. Everyone who reads it is bound to learn some things and be reminded of some things they’ve forgotten.

I was looking at host-based change discovery tools like aide and tripwire, and tested them out, but unfortunately I can’t afford the CPU cycles and hard drive bandwidth for an extra full read of the disks every night. However, it just occurred to me that the since I use incremental rsnapshot backups, I’m already checking the entire filesystem for changes. To determine the file integrity I simply have to find the difference between the most recent backup against the next oldest. By running this on the backup server I’m both offloading CPU cycles to a box that they’re less precious on, and also running the tests in a more secure environment.

Rsync normally checks file attributes to determine if there was a change, so a determined hacker could prevent rsync from picking up the change by making sure that the file size, creation, and modification times are the same. To be a functional tripwire replacement, one would need to enable –checksum in rsync_args, which causes rsync to physically checksum every file (and slow down the backup substantially). However, if you’re a little less paranoid and are interested in a general “change notification” scheme, than attribute-based difference detection is probably sufficient.

It turns out that rsnapshot already ships rsnapshot-diff, a tool for determining the difference between snapshots. It’s not quite suited for my application, but I should be able to whip up a wrapper pretty quickly that comes out with a useful file integrity report. I’ll post the script here when I make some progress.

My goals for upgrading to PHP 5 are to do it without breaking any customer websites, and also to use few or no legacy options. Hopefully by being strict about legacy options I can prevent pidgin-php that requires a combination of current and depreciated features, as that would make the upgrade to PHP 6 much more of a nightmare. Specifically url_fopen and register_globals will be turned off, and zend v1 compatibility mode will not be turned on. I have decided to leave register_long_arrays on, because of the sheer number of scripts that use $HTTP_*_VARS style formatting, and because there isn’t much harm in that syntax being available. Register_long_vars will be removed from php 6, but I’m willing to deal with that problem when we start looking at PHP 6 (fortunately, a file search to find all of the scripts using that syntax will be trivial).

Because of these substantial changes, I need a way for my customers to preview their websites rendered by the new php, so we can both be confident that when it comes time to flip the switch there won’t be any issues. I’ve done this by installing a new php and apache on an alternate port, and we’ll be emailing customers shortly to have them check their websites on the alternate port to verify they work correctly, and to make support available to customers who aren’t sure about how to fix their pages ( I actually anticipate very few will call).

I’d like to be able to include links that are not pages in my pages menu, as most themes display the pages menu as the main navigation bar for the site. The general wisdom has been to hard-code those links into the theme, however I want to be able to intersperse links
with pages, and I want to be able to switch themes without patching all of the themes I use.

I accomplished this with a plugin that takes an array of html to insert at points in the pages menu. For example, in my blog I have a link for “computers” and a link for “other thoughts”, which are before any of the real pages are displayed. The configuration line of my plugin looks like the following:

12. //An array of items to insert. 0 makes it the first link, 2 makes the link appear after link 2, etc.
13. $insert_array = array (
14. 0 => '<li><a href="http://erek.blumenthals.com/blog/category/linux" title="linux">Computers</a></li>',
15. 1 => '<li><a href="http://erek.blumenthals.com/blog/category/other" title="other">Other Thoughts</a></li>'
16. Download this code: addlinksaspages.php

To install this in your blog: download the file, customize the insert array with the links you’d like to add, drop the php file in your wp-content/plugins folder, and activate the plugin. Please drop me a line if you decide to use it, as I’m curious what people will do with it. Also feel free to shoot suggestions my way for improvements.

I’ve always been wary of automatic updates. Certainly, they give you the quickest response time, but at the price of stability. I’d rather know exactly what I’m updating when I’m updating it, so if I run into an issue it’s easier to backtrack where it came from. I find that a happy medium is email notification when updates become available. Since most of my servers are running the same distribution, I fire up clusterssh and update them all at once.

Here’s my yum email notification script (adapted from a script by Michael Heiming):

1. #!/bin/sh
2. maila="email@address.com"
3. yumdat="/tmp/yum-check-update.$$"
4. yumb="/usr/bin/yum"
5. CHECKWRK='no'
6. #rm -f ${yumdat%%[0-9]*}*
7.  
8. $yumb check-update >& $yumdat
9.  
10. yumstatus="$?"
11.  
12. case $yumstatus in
13. 100)
14. cat $yumdat |\
15. mail -s "Alert ${HOSTNAME} updates available!" $maila
16. exit 0
17. ;;
18. 0)
19. # Only send mail if debug is turned on
20. if [ ${CHECKWRK} = "yes" ];then
21. cat $yumdat |\
22. mail -s "Yum ${HOSTNAME} no patches available." $maila
23. fi
24. exit 0
25. ;;
26. *)
27. # Unexpected yum return status
28. (echo "Undefined, yum return status: ${yumstatus}" && \
29. [ -e "${yumdat}" ] && cat "${yumdat}" )|\
30. mail -s "Alert ${HOSTNAME} problems running yum." $maila
31. esac
32.  
33. # clean up
34.  
35. [ -e "${yumdat}" ] && rm ${yumdat}
36. Download this code: yumcheckupdates.cron

One more thing is that it’s easy to forget to reboot after a kernel update, or put off rebooting because the time you update the RPMs isn’t the best time to update the kernel. To make sure that I do reboot, I’ve written added an email notification script that lets me know when the latest installed kernel is newer than the currently running kernel:

1. #!/bin/sh
2. CONTACTADDR=email@domain.com
3. LATESTKERNEL=`rpm -q kernel |tail -n1|sed -e 's/kernel-//'`
4. if uname -a | grep -qv $LATESTKERNEL; then
5. echo "Running Kernel is" `uname -r` "but latest installed rpm is ${LATESTKERNEL}" |\
6. mail -s "${HOSTNAME} Reboot required" $CONTACTADDR
7. fi;
8. Download this code: check-kernel.cron

I’ll admit it. Like many other hosting companies, we’ve been remiss in upgrading to PHP 5 on our shared hosting servers. Apparently, we’re in good company. Nexen Services queried approximately 2 million servers and found the following (as of November 2007):

There may be some valid reasons for sticking with PHP 4, especially in highly controlled environments (think Yahoo), or in redistributable software, where you need to be able to reach the largest installed base without requiring architecture changes. However, as a web host to customers in non-controlled environments, and a developer who develops primarily within our own hosting environment, we have the luxury of being a cautious leader.

Also, I think that part of the reluctance to switch to PHP-5 is a fear of the new coding styles that it strongly encourages. The days of <?php include(’header.inc’) ?> are coming to an end, and are being replaced with MVC-esque coding structures. The general PHP community is taking time to come to terms with this.

It’s funny how realization of the need for change often comes out of a personal inconvenience. Last week I set out to write a substantial application to use on our hosting servers, developing it on my personal development box. I uploaded the code expectantly, and found that nothing worked. I had forgotten that PHP 4 lacks many of the magic object methods I’ve grown accustomed to using (__get, and __set, and __construct primarily).

I can either spend a day refactoring my code to work on PHP 4 (and make it much less elegant in the process) or I can take the plunge and get the ball rolling to upgrade all of our boxes to PHP 5. I’m going with the latter option. Implementation details to follow.