All about the system administration and application development behind a local linux-based company
I just finished a presentation at the Niagara Frontier LUG on Xen virtualization and how it applies to a managed hosting infrastructure. Here is a copy of the presentation for anyone who’s interested (sorry, it’s MS powerpoint. Yes, I do appreciate the irony).
It goes over some of the pros and cons of all virtualization, different types of virtualization, strategies for achieving levels of availability, and finally, some xen-specific configuration options and tips.
Edit: Mark asked me to clarify the benchmark parameters, so here’s an excpert of an email that I sent to another reader about this.
as an FYI, the benchmarks were done with:
core 2 duo 1.8Ghz
single 7200RPM 2.5″ driveThe image was done as a .img file on the same partition as the DomU, and the
partition was a separate DOS partition on the same drive. The only VMWare test
I ran used an image file. All of the OS installs were cloned from the same directory tree
(stay tuned for a future post on converting a Xen image to vmware)The Moodle benchmarks were a snapshot of a production moodle install,
where I copied the install to my test system, logged in and clicked a
few things, and replayed the session logs through jmeter several hundred
times with five concurrent threads.The images benchmark is transfer speed of randomly selecting 1 of 2500
10kb-40kb images. 8 concurrent users and 1500 iterations.The Mysql benchmark is the results from running all the tests in the
mysql benchmark suite.Of note, the image file generally performed a little bit better than the
raw partition. This is counter to what the Xen documentation and common
sense would say, and I think a lot of it has to do with my pretty
limited tests. The image file was ending up in memory cache, whereas
the block device wasn’t. I doubt the same comparitive performance would
play out in a production system where a lot more’s going on.
I told myself when I started this blog that I was going to avoid simple howto posts and focus on system administration concepts. However, I’m going to break that rule today. I was looing through my referrer log files, and noticing that I get a lot of search queries for “How to…” or “how doI…” So, I dug up my access logs, fired up egrep, and made myself a list of search referrals with “how” in the string. Today I’m going to give in to the masses and attempt to answer some of them :
Clusterssh and shmux do this, as mentioned in a previous post. For a more scalable system, also have a look at puppet.
run “yum update” to update all of your software packages. Be sure to restart after a kernel update.
Tripwire is a commercial product, so RTFM or bug support. Also see aide for a free version with similar design goals.
Do you really want to do this? Might ssh/scp/rsync over ssh be a better option? The only possible reason is for speed in an isolated network, where every machine on the network is trusted.
Install the packages rsh-server and xinetd. edit /etc/xinetd.d/rsh and change disable = yes to disable = no. Add .rhosts files as needed. To enable rlogin, do the same for /etc/xinetd.d/rlogin
It isn’t installed by default. If it is installed, edit /etc/xinetd.d/rsh, /etc/xinetd.d/rlogin, and /etc/xinetd.d/rcp and add “disable = yes” to all of them. Or, just remove the whole package with “yum erase rsh”
Run yum check-update , intuitively enough
If you must, install the telnet-server package and enable the service in /etc/xinetd.d/telnet
Thanks to Andrew for the suggestion to add a plug for ssh here.
I assume you mean how to disable root ssh logins. open /etc/ssh/sshd_config and add a line “PermitRootLogin no” Addition: Also, set more restrictive permissions to /bin/su so that people can’t log into ssh as a normal user and su to root.
Since the old kernel should still be installed, simply edit /etc/grub.conf and change default=0 into default=1 Be sure to change it back when you get the kernel working, or you will perpetually be running one kernel behind current
See my post on getting email notifications which I think is a better strategy. If you really must do automatic updates, create a daily crontab that runs yum -y update.
pidgin.
Apple’s (in)famous for using somewhat bizarre internal codenames for their projects. Looks like they haven’t let us down with the iPhone SDK, apparently previously codenamed SquarePants. I just went to change my Apple password so I could download the iPhone SDK, and after I was finished I came across the following message:
Your Apple ID password has been changed
Select the “continue” button below to return to SquarePants Web Portal
Remember to change your internal product names everywhere they display before you bring a web application live. If they must be hard coded, put them in as few places as possible, or even better in a configuration system (like that ships with Zend Framework) that allows you to set development and production parameters separately.
Anyway, here’s a screenshot:
It often comes up where I’ve got a mail delivery issue (user n can’t send email to domain n.com) and it’s difficult to troubleshoot just with the info in the logs. One thing that sometimes yields useful info is directly telnetting to port 25 of the offending mail server. This way if it’s rejecting messages you can see the exact reject message, or it could be violating the protocol in some creative way.
I just ran into a succinct protocol description at http://helpdesk.islandnet.com/pep/smtp.php
Here’s their transcipt of a successful SMTP session. Be sure to notice the excra line break between the end of the headers and the beginning of the body.
telnet mail.islandnet.com 25
220 Islandnet.com ESMTP server ready
helo a.b.c
250 mail.islandnet.com Hello x [YOUR_IP_ADDRESS]
mail from:
250is syntactically correct
rcpt to:
250verified
data
354 Enter message, ending with “.” on a line by itself
From: Bugs Bunny
To: Daffy Duck
Subject: Loony Toons!Hi there!
.
250 OK id=1778te-0009TT-00
quit
221 mail.islandnet.com closing connection
Well, it’s all over with. I hope that everyone enjoyed the wild patching frenzy. Make sure you update your kernel to the CentOS-supported one at your earliest convenience. I’ll remove all the kernels except for the ones from the 53.1.4 tree soon to avoid confusion. Many thanks to everyone in the Redhat and CentOS teams for their quick and diligent responses.
From Centos-Announce
The following updated files have been uploaded and are currently
syncing to the mirrors:i386:
kernel-2.6.18-53.1.13.el5.i686.rpm
kernel-devel-2.6.18-53.1.13.el5.i686.rpm
kernel-doc-2.6.18-53.1.13.el5.noarch.rpm
kernel-headers-2.6.18-53.1.13.el5.i386.rpm
kernel-PAE-2.6.18-53.1.13.el5.i686.rpm
kernel-PAE-devel-2.6.18-53.1.13.el5.i686.rpm
kernel-xen-2.6.18-53.1.13.el5.i686.rpm
kernel-xen-devel-2.6.18-53.1.13.el5.i686.rpm
I’ve finished compiling kernels based on the 2.6.18-53.1.4 CentOS kernel. Some people are still using it, because the latest RHEL/CentOS kernel can have NFS issues.
The kernels are christened 2.6.18-53.1.5.el5.cve20080600. They’re available at http://erek.blumenthals.com/vmsplicekernels/ .
Keep in mind that they’re entirely untested, as I haven’t had a chance to go through each one. Please download them, do your own QA, and comment or email me with the results.
Redhat has released updated RPMs for RHEL 5.1 uncharacteristically quickly, in recognition of the seriousness and internet coverage of the issue: RHSA-2008-0129. I expect we’ll see a release from centos soon as well. Of note, this release does not fix the nfs issues that were present in 2.6.18-53.1.6.
At the suggestion of a Centos mailing list member, I’ll be posting RPMs from the 2.6.18-53.1.4 release soon, for people who need to run the earlier version because of NFS issues.
I’ve built the following RPMs for RHEL 5 that fix the vmsplice() exploit in RHEL machines. They are built off of the 2.6.18-53.1.6.el kernel, with the upstream patch from kernel.org.
I’ve tested them on i686 and x86_64 machines, however be aware that they have not undergone extensive QA, so I’m not responsible if they blow up your machine. That said, I’m pretty confident that no one will have any problems with them, as they are literally a one-line difference.
Update: Reminder to install these with rpm -ivh and not rpm -Uvh. Otherwise you’ll remove your old kernels, which you may need to fall back to,.
i686:
i688-PAE:
x86_64:
Source:
Xen, and several other RPMs are available at: erek.blumenthals.com/vmsplicekernels. Note that the PAE and Xen kernels are entirely untested.
As this kernel is an odd-numbered release, yum should pick up the official upstream patch as soon as it’s available, but in case they do their numbering differently or pick the same release number that I did, it’d be a good idea to double check that yum picks up the latest.
Let me know any experiences with this, especially any confirmations that it’s safe with PAE or Xen.
See http://www.milw0rm.com/exploits/5092 for proof of concept code.
I’ve verified this to work:
[erek@centosmachine src]$ uname -a [erek@centosmachine src]$ ./exploit ----------------------------------- Linux vmsplice Local Root Exploit By qaaz ----------------------------------- [+] mmap: 0x0 .. 0x1000 [+] page: 0x0 [+] page: 0x20 [+] mmap: 0x4000 .. 0x5000 [+] page: 0x4000 [+] page: 0x4020 [+] mmap: 0x1000 .. 0x2000 [+] page: 0x1000 [+] mmap: 0xb7fad000 .. 0xb7fdf000 [+] root [root@centos5machine src]# whoami root [root@centos5machine src]#
Ubuntu, Centos 5, and most Fedoras seem to be vulnerable. Centos 4 is not. I’m recompiling Centos 5 and FC 3 kernel RPMs with the appropriate patches, and will post them here in an hour or two. These are using the upstream kernel patch and I’ll know soon whether they conflict with any of the RHEL-specfic code. I doubt it does, as it’s a one-line patch.
And that’s the sound of 1000 admins running home from their Sunday afternoons to patch their boxes, and the sound of 1000 cell phones going off as their bosses read about this.
Update: Compiler is still going, and I’m heading out. I’ll post the rpms in the morning.