I’ve been pretty good at keeping my server up to date. I’ve taken a general policy of “if it isn’t broke, don’t touch it”. It works well, for the most part. When it comes to servers though, security is a good idea, and that plays havoc with the previous mantra.
Warning: This post is long… not only is it long, but I’m pretty sure it’ll be filled with profanities, and general obscene guestures, and words… Read on if you dare…
For example, this evening, I noticed a new security update required on my server.
# glsa-check --list | grep '\[N\]'
[A] means this GLSA was already applied,
[U] means the system is not affected and
[N] indicates that the system might be affected.
200611-26 [N] ProFTPD: Remote execution of
arbitrary code ( net-ftp/proftpd )
200702-02 [N] ProFTPD: Local privilege
escalation ( net-ftp/proftpd )
Okay, both for the same package, that’s not too bad, but there must have been a reason I didn’t do the first one earlier, it was dated November. So I see what it requires…
# emerge -pv proftpd
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild U ] dev-db/mysql-5.0.26-r2 [4.1.22-r1]
[ebuild N ] virtual/mysql-5.0 0 kB
[ebuild U ] net-ftp/proftpd-1.3.1_rc2 [1.2.10-r7]
Total: 3 packages (2 upgrades, 1 new), Size of downloads: 1,481 kB
Well there is the reason. Upgrading MySQL from 4.0 to 5.0. So I try to cheat, and see if I can get away with an upgrade from MySQL 4.0 to 4.1. Note, there are some caveats to this upgrade. So I do the upgrade, and attempt to restart MySQL… Horror strikes, as you can imagine, otherwise I wouldn’t be posting this.
070302 2:21:47 [ERROR] Fatal error: mysql.user table is damaged
or in unsupported 3.20 format.
Ouch, this shouldn’t happen. Then I remember some of the caveats, and realize the permissions tables have to be upgraded. Something you’d naturally hope the installer to handle, but oh well. So I attempt to run the update script mysql_fix_privilege_tables, only to be confronted with the error that it couldn’t connect. Of course not, MySQL wouldn’t start, so what was I thinking. So something wrong with the privs table, and you cannot start MySQL, this is a kicker. It is actually easy to solve. It’s the same solution you use to start MySQL if you’ve forgotten the password for root.
# mysqld_safe --skip-grant-tables --user=root &
You’ll get a nice little blurb about it starting, and then you can run the mysql_fix_privilege_tables command again.
Now realizing a major version upgrade of a database service is a big task, and the chances of it touching other applications is actually pretty high, so I start running down the list of services, and see what is impacted.
- Webservices… OK
- Email services (IMAP) … OK
- Email services (POP3)… OK
- Email services (SMTP)… DEAD
Crap. My Postfix is built with MySQL for virtual hosting capabilities… So I stop and start it, but it refuses to start, so I skip the use of the init script, and try starting it manually…
postfix: error while loading shared libraries: libmysqlclient.so.12: cannot open shared object file: No such file or directory
Of course, why would an upgrade of one service go so smoothly. So the libraries have changed names, and are probably not compatible with the so.12 libraries, which means a simple link might not work. So off I go to see what a rebuild of Postfix will required.
# emerge -pv postfix
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild U ] dev-db/mysql-5.0.26-r2 [4.1.22-r1]
[ebuild N ] virtual/mysql-5.0 0 kB
[ebuild U ] mail-mta/postfix-2.3.6 [2.1.5-r2]
Total: 3 packages (2 upgrades, 1 new), Size of downloads: 2,719 kB
What? Oh dear, they want to upgrade MySQL too. So I try to cheat, and use the same version of Postfix I already have installed, by using version specific installs:
# emerge -pv "=postfix-2.1.5-r2"
It still wants MySQL 5, which means they probably have something in the ebuild that just requests the lasts MySQL version… sighs. So the jump to MySQL 5.0 shouldn’t be too bad, I’ve just hurdled the 4.1 jump… I’ve already verified most of my apps work on MySQL 5 on my dev server, so this is just a little jump.
While MySQL 5 is building, I start to wonder what else I have using MySQL, and stumble across maildrop. I decide to confirm if it really needs a rebuild as I had to make some custom changes on this…
# ldd /usr/bin/maildrop
linux-gate.so.1 => (0xffffe000)
libgdbm.so.3 => /usr/lib/libgdbm.so.3 (0xb7fdc000)
libmysqlclient.so.12 => not found
libz.so.1 => /lib/libz.so.1 (0xb7fc4000)
libcrypt.so.1 => /lib/libcrypt.so.1 (0xb7f96000)
libnsl.so.1 => /lib/libnsl.so.1 (0xb7f81000)
libssl.so.0.9.7 => /usr/lib/libssl.so.0.9.7 (0xb7f45000)
libcrypto.so.0.9.7 => /usr/lib/libcrypto.so.0.9.7 (0xb7e1e000)
libstdc++.so.5 => /usr/lib/gcc-lib/i686-pc-linux-gnu/libstdc++.so.5
libm.so.6 => /lib/libm.so.6 (0xb7d2e000)
libgcc_s.so.1 => /usr/lib/gcc-lib/i686-pc-linux-gnu/libgcc_s.so.1
libc.so.6 => /lib/libc.so.6 (0xb7c11000)
libdl.so.2 => /lib/libdl.so.2 (0xb7c0d000)
/lib/ld-linux.so.2 (0xb7feb000)
Drats, it’s build against libmysqlclient.so.12 which means I’m going to have to rebuild this too. Time to take a look at the source code for this. I sure wish Erin was awake, she helped fix the original version… So checking on the latest version of maildrop… I was in luck. They’ve dropped mysql support, in favour of authentication through Courier’s authlibs. This is good, because I know that my imap services are still running, though I’ve not tested them yet. Eagerly I rebuilding maildrop, with fingers crossed…
I should have stayed in bed..
Postfix started throwing out errors…
postfix/pipe[13656]: 4B61D125F2B: to=<address>,
relay=maildrop, delay=2.5, delays=0.21/0.38/0/1.9, dsn=5.1.1,
status=bounced (user unknown. Command output: Invalid user
specified. )
What on earth? Now I remember these from when I originally setup the server, and it took me a while to resovle them. So I start googling around. This is a pretty common error because of people’s setups, but I figure I should try some debugging to begin…
# maildrop -V 9 -d <address>
Invalid user specified.
Hrm, so it thinks there are no valid users, then it strikes me. The email address being fed here is an alias, and not my username. Maildrop is expecting my username. So why is postfix feeding maildrop the alias. Then it dawns on me to check how I have courier-authlib configured, and how I had maildrop configured. There is one minor difference. The old maildrop configurations looked at the email column, while courier looks at the username column. Bingo. So I do a quick update of my email column, and add a real alias record for my name, and all is working again…
So with postfix updated, and a minor screw up with etc-update, and accidentally overwriting my configuration file (my fault), I set out to check other applications. Lets restart apache…
Cannot load /usr/lib/apache2/modules/libphp4.so into
server: libmysqlclient.so.12: cannot open shared
object file: No such file or directory
Argh!!! So off to emerge PHP again. This is becoming one hell of a nightmare… Shortly followed by this one when trying to emerge php…
TSRM/tsrm.m4:71: file `TSRM/TSRM/threads.m4' does not exist
It’s not 4:42am, and it doesn’t look like I’m going to bed tonight. I start looking around for some solutions, and the first page I found was in german, but had no solution to it, so I hit the Gentoo forums, which thankfully returned this little gem. It basically suggests upgrading aclocal-1.9. So I checked, and mine was at 1.9.4, so I pushed another update on the list:
emerge =sys-devel/automake-1.9*
When will these updates end…
Apparently not any time today…
# /etc/init.d/courier-authlib restart
* Stopping courier-imapd... [ ok ]
* Stopping courier-imapd over SSL... [ ok ]
* Stopping courier-pop3d... [ ok ]
* Stopping courier-pop3d over SSL... [ ok ]
* Stopping courier-authlib: authdaemond... [ ok ]
* Starting courier-authlib: authdaemond... [ ok ]
* Starting courier-imapd... [ ok ]
* Starting courier-imapd over SSL... [ ok ]
* Starting courier-pop3d... [ ok ]
* Starting courier-pop3d over SSL... [ ok ]
That was relatively painless, or so I thought. In the background, my thunderbird started blinking, and asking for the username and password. Not a good sign I must say… especially not as it is now rolling around to 05:30am… Back to the log files, and this item hits me again.
authdaemond: libmysqlclient.so.12:
cannot open shared object file: No such file or directory
Crap again. Not only is courier broken now, but it has nicely gone and broken maildrop because it was using it for authentication… At least its an easy fix, upgrade courier. Fortunately enough, it’s only one minor version behind due to a recent security issue…
# emerge courier-imap courier-authlib
A quick restart of all the services, and everything was back working again, including maildrop.
During all of this “hickups”, I completely forgot the original goal… To upgrade proftpd. This one should be easy, I’ve already fixed the requirement for MySQL 5 (which I think is somewhere in some Gentoo Dev’s head). So a quick emerge proftpd and a restart should be fine. Fortunately, this was the easiest upgrade of the evening.
So far, everything seems to be running okay… I’m hoping it stays this way… Now it’s 6:45, so I’m going to have a shower and get ready for work…
A final note… If you ever look at something and wonder if it’s worth doing… the answer is inevitably NO. If you have to think for a second… it’s no… always.