Small linux deployments

Read the rest of this entry »

I told myself when I started this blog that I was going to avoid simple howto posts and focus on system administration concepts. However, I’m going to break that rule today. I was looing through my referrer log files, and noticing that I get a lot of search queries for “How to…” or “how doI…” So, I dug up my access logs, fired up egrep, and made myself a list of search referrals with “how” in the string. Today I’m going to give in to the masses and attempt to answer some of them :

Linux how to execute simultaneous commands? how to run same command on multiple servers?

Clusterssh and shmux do this, as mentioned in a previous post. For a more scalable system, also have a look at puppet.

How to download 2.6.18-53.1.13 rhel?

run “yum update” to update all of your software packages. Be sure to restart after a kernel update.

Centos 5 : how to install tripwire?

Tripwire is a commercial product, so RTFM or bug support. Also see aide for a free version with similar design goals.

how do i enable rsh on rhel5?, how to configure rlogin in rhel5

Do you really want to do this? Might ssh/scp/rsync over ssh be a better option? The only possible reason is for speed in an isolated network, where every machine on the network is trusted.

Install the packages rsh-server and xinetd. edit /etc/xinetd.d/rsh and change disable = yes to disable = no. Add .rhosts files as needed. To enable rlogin, do the same for /etc/xinetd.d/rlogin

how to deactivate rsh on rhel 5

It isn’t installed by default. If it is installed, edit /etc/xinetd.d/rsh, /etc/xinetd.d/rlogin, and /etc/xinetd.d/rcp and add “disable = yes” to all of them. Or, just remove the whole package with “yum erase rsh”

how to check for kernel update rhel5

Run yum check-update , intuitively enough

How to deploy database in mysql in linux

1. Remove all flammable objects from the room.
2. Lift the protective clear plastic cover and push the “deploy database” button.
3. Take cover.

how to enable telnet in rhel5? how to install telnet on rhel5? how to enable the telnet on the rhel5 server?

If you must, install the telnet-server package and enable the service in /etc/xinetd.d/telnet

Thanks to Andrew for the suggestion to add a plug for ssh here.

how to restrict other people from rebooting in linux?

1. Remove ca::ctrlaltdel:/sbin/shutdown -t3 -r now from /etc/inittab , or add a -a option to that line and put a list of users allowed to shut the system down in /etc/shutdown.allow
2. edit /etc/gdm/custom.conf and add the line SystemMenu = false to the [greeter] section

How to restrict root logins to the system console?

I assume you mean how to disable root ssh logins. open /etc/ssh/sshd_config and add a line “PermitRootLogin no” Addition: Also, set more restrictive permissions to /bin/su so that people can’t log into ssh as a normal user and su to root.

How to roll back centos kernel?

Since the old kernel should still be installed, simply edit /etc/grub.conf and change default=0 into default=1 Be sure to change it back when you get the kernel working, or you will perpetually be running one kernel behind current

how to set automatic updates on rhel5?

See my post on getting email notifications which I think is a better strategy. If you really must do automatic updates, create a daily crontab that runs yum -y update.

how to use yahoo messenger in linux el 5

pidgin.

and the junk bin…

• how to fix a mail size for sendmail in linux el5
• how to exploit apache serveur
• php mysqli href how to a link data from one side to another

Apple’s (in)famous for using somewhat bizarre internal codenames for their projects. Looks like they haven’t let us down with the iPhone SDK, apparently previously codenamed SquarePants. I just went to change my Apple password so I could download the iPhone SDK, and after I was finished I came across the following message:

Your Apple ID password has been changed

Select the “continue” button below to return to SquarePants Web Portal

Remember to change your internal product names everywhere they display before you bring a web application live. If they must be hard coded, put them in as few places as possible, or even better in a configuration system (like that ships with Zend Framework) that allows you to set development and production parameters separately.

Anyway, here’s a screenshot:

(click for full size)

It often comes up where I’ve got a mail delivery issue (user n can’t send email to domain n.com) and it’s difficult to troubleshoot just with the info in the logs. One thing that sometimes yields useful info is directly telnetting to port 25 of the offending mail server. This way if it’s rejecting messages you can see the exact reject message, or it could be violating the protocol in some creative way.

I just ran into a succinct protocol description at http://helpdesk.islandnet.com/pep/smtp.php

Here’s their transcipt of a successful SMTP session. Be sure to notice the excra line break between the end of the headers and the beginning of the body.

telnet mail.islandnet.com 25
220 Islandnet.com ESMTP server ready
helo a.b.c
250 mail.islandnet.com Hello x [YOUR_IP_ADDRESS]
mail from:
250 is syntactically correct
rcpt to:
250 verified
data
354 Enter message, ending with “.” on a line by itself
From: Bugs Bunny
To: Daffy Duck
Subject: Loony Toons!

Hi there!
.
250 OK id=1778te-0009TT-00
quit
221 mail.islandnet.com closing connection

Well, it’s all over with. I hope that everyone enjoyed the wild patching frenzy. Make sure you update your kernel to the CentOS-supported one at your earliest convenience. I’ll remove all the kernels except for the ones from the 53.1.4 tree soon to avoid confusion. Many thanks to everyone in the Redhat and CentOS teams for their quick and diligent responses.

From Centos-Announce

The following updated files have been uploaded and are currently
syncing to the mirrors:

i386:
kernel-2.6.18-53.1.13.el5.i686.rpm
kernel-devel-2.6.18-53.1.13.el5.i686.rpm
kernel-doc-2.6.18-53.1.13.el5.noarch.rpm
kernel-headers-2.6.18-53.1.13.el5.i386.rpm
kernel-PAE-2.6.18-53.1.13.el5.i686.rpm
kernel-PAE-devel-2.6.18-53.1.13.el5.i686.rpm
kernel-xen-2.6.18-53.1.13.el5.i686.rpm
kernel-xen-devel-2.6.18-53.1.13.el5.i686.rpm

I’ve finished compiling kernels based on the 2.6.18-53.1.4 CentOS kernel. Some people are still using it, because the latest RHEL/CentOS kernel can have NFS issues.

The kernels are christened 2.6.18-53.1.5.el5.cve20080600. They’re available at http://erek.blumenthals.com/vmsplicekernels/ .

Keep in mind that they’re entirely untested, as I haven’t had a chance to go through each one. Please download them, do your own QA, and comment or email me with the results.

Redhat has released updated RPMs for RHEL 5.1 uncharacteristically quickly, in recognition of the seriousness and internet coverage of the issue: RHSA-2008-0129. I expect we’ll see a release from centos soon as well. Of note, this release does not fix the nfs issues that were present in 2.6.18-53.1.6.

At the suggestion of a Centos mailing list member, I’ll be posting RPMs from the 2.6.18-53.1.4 release soon, for people who need to run the earlier version because of NFS issues.

I’ve built the following RPMs for RHEL 5 that fix the vmsplice() exploit in RHEL machines. They are built off of the 2.6.18-53.1.6.el kernel, with the upstream patch from kernel.org.

I’ve tested them on i686 and x86_64 machines, however be aware that they have not undergone extensive QA, so I’m not responsible if they blow up your machine. That said, I’m pretty confident that no one will have any problems with them, as they are literally a one-line difference.

Update: Reminder to install these with rpm -ivh and not rpm -Uvh. Otherwise you’ll remove your old kernels, which you may need to fall back to,.

i686:

• kernel-2.6.18-53.1.7.el5.erek.i686.rpm
• kernel-devel-2.6.18-53.1.7.el5.erek.i686.rpm

i688-PAE:

• kernel-PAE-2.6.18-53.1.7.el5.erek.i686.rpm
• kernel-PAE-devel-2.6.18-53.1.7.el5.erek.i686.rpm

x86_64:

• kernel-2.6.18-53.1.7.el5.erek.x86_64.rpm
• kernel-devel-2.6.18-53.1.7.el5.erek.x86_64.rpm

Source:

• kernel-2.6.18-53.1.7.el5.erek.src.rpm

Xen, and several other RPMs are available at: erek.blumenthals.com/vmsplicekernels. Note that the PAE and Xen kernels are entirely untested.

As this kernel is an odd-numbered release, yum should pick up the official upstream patch as soon as it’s available, but in case they do their numbering differently or pick the same release number that I did, it’d be a good idea to double check that yum picks up the latest.

Let me know any experiences with this, especially any confirmations that it’s safe with PAE or Xen.

See http://www.milw0rm.com/exploits/5092 for proof of concept code.

I’ve verified this to work:

[erek@centosmachine src]$ uname -a

[erek@centosmachine src]$ ./exploit
-----------------------------------
 Linux vmsplice Local Root Exploit
 By qaaz
-----------------------------------
[+] mmap: 0x0 .. 0x1000
[+] page: 0x0
[+] page: 0x20
[+] mmap: 0x4000 .. 0x5000
[+] page: 0x4000
[+] page: 0x4020
[+] mmap: 0x1000 .. 0x2000
[+] page: 0x1000
[+] mmap: 0xb7fad000 .. 0xb7fdf000
[+] root
[root@centos5machine src]# whoami
root
[root@centos5machine src]#

Ubuntu, Centos 5, and most Fedoras seem to be vulnerable. Centos 4 is not. I’m recompiling Centos 5 and FC 3 kernel RPMs with the appropriate patches, and will post them here in an hour or two. These are using the upstream kernel patch and I’ll know soon whether they conflict with any of the RHEL-specfic code. I doubt it does, as it’s a one-line patch.

And that’s the sound of 1000 admins running home from their Sunday afternoons to patch their boxes, and the sound of 1000 cell phones going off as their bosses read about this.

Update: Compiler is still going, and I’m heading out. I’ll post the rpms in the morning.

Some open source applications (ex. Request Tracker) have excellent practices when it comes to their database schemas, but a surprising number of them do not.

This probably stems from many open source authors being programmers first and DBAs second, so the procedural logic is closer to their design decisions than the database schemas. The other day, when I was reading the installation manual for a popular web calendar system, I came across the following:

Next, create the database user account
that will be used to access the database.
mysql –user=root mysql
mysql> GRANT ALL PRIVILEGES ON *.*
TO webcalendar@localhost
IDENTIFIED BY ‘webcal01′
WITH GRANT OPTION;
mysql> FLUSH PRIVILEGES;
mysql> QUIT

Obviously giving root mysql access to a single web application is a grave mistake. This shows an absolute lack of the understanding of database administration.

So, with this example fresh in my head, I’m going to go through some practices that I’ve found to be helpful for maintaining extensible database schemas. As usual, the examples will be tailored to MySQL >4.1 and PHP 5, but there’s no reason they couldn’t be adapted to other languages. I’ve optimized these examples for robustness and auditability, without a lot of regard for disk usage. Frankly disk space is quite cheap, and throwing away information is rarely the best way to go.

This is in no particular order:

INSERT data, rather than UPDATEing, and UPDATE rather than DELETEing

If you’re building a CMS, when a user modifies a page, you can either add a new row for the modified page, or you can UPDATE the current row. The difference is that inserting a second row makes rolling back the old page easy, and also allows you to generate a log straight from the database (”User “y” modified page “z”. Diff: …). Of course, this makes the SELECTs a little bit uglier:

SELECT pagename, content, title FROM pages
WHERE pagename = "foo" ORDER BY timestamp DESC LIMIT 1
versus
SELECT pagename, content, title FROM pages
WHERE pagename = "foo"

However, he first option is auditable and rollback-ready, and with a few simple modifications could allow administrator approvals, etc. The second option requires the user who edits the page to get it right the first time.

Similiarly, if your goal is to remove a user, you can either DELETE that user’s row, or you can add a boolean Disabled field, and UPDATE them to Disabled instead. This way you don’t have to be nearly as careful with referential integrity, as the user’s primary key still exists,and thus any tables that reference that id will stay consistent.

When extending your application, add fields and tables, rather than altering existing ones

If you add a new tables and fields as your functionality grows, rather than modifying the older ones, you can roll back the application to the previous version, and it will
still work as it always did, since all of the data it expects will be in the form it was. Over time this can result in somewhat bloated database schemas, but good documentation and
cleanup can mitigate this.

Enumerate all of the fields you expect will be returned, rather than using wildcards

SELECT * from foo;
May break your application when you add extra fields to the database, or if the order of the fields changes, however:
SELECT field1, field2 from foo;
will continue to work even as the database schema evolves.

An alternate way of accomplishing the same design goal is to use a wildcard in the SQL but access the fields by name in your application code. For example:

$result = $mysqli->query('SELECT * from foo;');
$foo = $result->fetch_assoc;
do_stuff_with_fields1_and_2($foo['field1'], $foo['field2'];


This strategy results in even easier application extension, as you can immediately use new fields in your application logic without having to modify your SQL queries. The disadvantage is that you may be pulling more information out of your database than you plan to use, expending extra memory and loading down the DBMS.

Use prepared statements whenever possible

Here are two examples of PHP code that perform the same function (Error handling not included):


$searchstringescaped = $mysqiobject->escape_string($searchstring)
$result = $mysqli->query("SELECT foo, bar, bak WHERE bak like '$searchstringescaped'");
while(list($foo, $bar, $bak) = $result->fetch_row()) {
do_stuff_with_bar_and_bak()
}


Or, with prepared statements:

$stmt = $mysqli->prepare("SELECT foo, bar, bak WHERE bak like ?");
$stmt->bind_param('s', $searchstring);
$stmt->execute();
$stmt->bind_result($foo, $bar, $bak)
while($stmt->fetch()) {
do_stuff_with_bar_and_bak();
}


While the former may have fewer lines, the second one clearly separates the query logic from the parameters, and eliminates any possibility of SQL injection or double escaping.

Use Transactions

(This requires innodb)

Any database statement can fail. It can be because of programmer error, hardware failure, or any other host of issues. When executing a group of statements, it’s almost always better for all of them to fail than for only some of them to fail. For example, suppose that you want to transfer credits from one user’s account to another. The naive solution would be:

UPDATE users SET credits = credits + 1000 WHERE id = 1;
UPDATE users SET credits = credits - 1000 WHERE id = 2;

However, this code has two problems. First of all, if there were a power failure at precisely the right time, it would result in both users having the thousand credits. Second, for a short amount of time, the credits would be in both accounts, and a third client viewing the accounts between the two updates would see inconsistent data. The proper way to do this would be:

START TRANSACTION;
UPDATE users SET credits = credits + 1000 WHERE id = 1;
UPDATE users SET credits = credits - 1000 WHERE id = 2;
COMMIT;

Or, if you’re using PHP 5’s Mysqli extension, you can use the extension’s interface to transactions like:

$mysqli->autocommit(false);
//Run some mysqli queries
$mysqli->commit();

Finally, if something in the business logic causes you to change your mind about running the transaction, you can call a rollback, which will cancel the last transaction:


$mysqli->autocommit(false);
$mysqli->query('UPDATE users SET credits = credits + 1000 WHERE id = 1;');
$mysqli->query('UPDATE users SET credits = credits - 1000 WHERE id = 2;');
$result = $mysqli->query('SELECT credits from users WHERE id = 2');
list($giverbalance) = $result->fetch_row();
if($giverbalance < 0) {
//Ooops, the giver now has a negative balance.
$mysqli->rollback();
}


I hope that this helps, in a small way, to further the practices of database developers, and I’d love to hear any more suggestions or comments people have on the topic.

Update (Feb 02 2007): Fixed several grammatical errors