I’ve been asked to write a post on configuration management, and version control by a friend, Steven Klassen. Instead of a single post, I’m going to break this into several posts as some parts might be unimportant to some, and they can easily skip a whole post. I’ll be posting this over a few days, so stay tuned. If you’re not subscribed to my RSS feed, now would be a great time.

The Request

Earlier today, Steven asked if I used any form of version control. It was a leading question, straight into “dev only or system files”? His questions were targeted, he was searching for guidance. Another question or two later, and it boils down to this:

• Do you use version control for system files?
• Files like domain zone files?
• What about multi-user environments?
• Can you write it up?

My take away on the questions boils down to two things; configuration management, and backups. Anybody that has managed any type of system has, at some point, messed up the configuration file, and forgotten what they did to get there in the first place. Those who have been at it a while will usually do a quick copy before they do any modifications. But, as Steven found out in the past, best laid plans can be foiled by a simple typo.

Version Control

This is what version control is all about. It acts as a backup, whilst keeping a running history of what you did between changes. Version control, or revision control as it’s also known, is big business. Anywhere you find a big name development company, you’ll find they’ve probably developed a revision control system to go with whatever they’re offering. Take a look at this Wikipedia entry comparing just a handful of them.

Configuration Management and Version Control

You don’t have to be a genius to figure out how useful version control can be for configuration management, but in case your brain often goes off to lala land like mine, here is a run down on a few reasons you should really consider using it…

• What did I just change again? And why is that service not starting any more? Oh hell…
• 6 months down the road, can you figure out why you changed that line to Apache to start 10 daemons instead of 5? No? Most version control systems allow you to add comments to your commits.
• As above, but why did Fred in your team change that entry back down to 7 a week later?
• In the words of Homer… Doh!! I wasn’t supposed to delete/overwrite that file!

So, there are 4 obvious cases as to why you’d want to use version control.

The Breakdown

As I said at the beginning, I’ll be breaking this down into several parts. I expect them to be something along these lines:

• Introduction
• Setting up SVN and the base repository
• Using your repository
• Educating your team members
• Maintenance and monitoring
• Conclusion, and follow-up ideas.

If you’re interested, keep an eye open for the rest of the series. If not, send me something you’d like to see me write about. I’m open to suggestions, and will probably butcher anything you can throw at me.

In a case of what I’d call security through obscurity, the Linux Poison blog has a posting up on how to hide the PHP information in your web server. The idea is that the less a potential hacker knows about a system, the better. Here, I’m going to take it one step further, and show you the impact of some other options, and what data gets hidden.

The first step is to see what information is displayed without making changes. From my development server, I pull up Firebug, and load a page, and see what the response headers look like…

As you can see from the image of Firebug there is a wealth of information being presented to any would-be hacker. To start, they now know my development server is running Debian Lenny. It also has mod_php, mod_perl, mod_ssl, and mod_python (plus specific version information) installed, and that the instance of apache running is using OpenSSL 0.9.8g. A quick search of any security tracking sites could pull up all kinds of useful information based around that alone. This just made the hackers life easier, because now s/he doesn’t have to brute force attack the server.

Lets see what happens when we modify the recommendations by the Linux Poison blog. In Debian the file needed to be modified is in /etc/php5/apache/php.ini, other platforms may store the file elsewhere. Don’t forget to tell your web server to reload after making the changes.

This looks a little better. It no longer displays any PHP related information, but the rest of the data is still there. We need to do something more about that. In my case, that means moving onto the apache configuration. The two important directives in this case are ServerTokens and ServerSignature. These directives are both in /etc/apache2/apache2.conf in Debian, so may be located in a different file depending on platform. ServerTokens is the most important one. A quick look at the apache documentation shows what the output looks like depending on the options selected…

ServerTokens Prod[uctOnly]
   Server sends (e.g.): Server: Apache
ServerTokens Major
   Server sends (e.g.): Server: Apache/2
ServerTokens Minor
   Server sends (e.g.): Server: Apache/2.0
ServerTokens Min[imal]
   Server sends (e.g.): Server: Apache/2.0.41
ServerTokens OS
   Server sends (e.g.): Server: Apache/2.0.41 (Unix)
ServerTokens Full (or not specified)
   Server sends (e.g.): Server: Apache/2.0.41 (Unix) PHP/4.2.2 MyMod/1.2

From this list, it’s clear to see that prod is the winner on showing the least amount of information. So a quick change of that setting, and a restart of apache again, lets see what happens to the output now…

As we can see from the Firebug output, it now lists only that I’m running apache, a substantial improvement.

What about the ServerSignature option I mentioned? The apache documentation has this to say about this configuration entry…

The ServerSignature directive allows the configuration of a trailing footer line under server-generated documents (error messages, mod_proxy ftp directory listings, mod_info output, …). The reason why you would want to enable such a footer line is that in a chain of proxies, the user often has no possibility to tell which of the chained servers actually produced a returned error message.

It mentions server-generated documents, so this includes any error pages, but it also includes the pages Apache generates when you don’t have a directory index, and it just lists the file contents. The option, and the ServerTokens option actually play together, so for a test, I’ve reverted the ServerTokens option back to Full so you can see what happens when ServerSignature is set to On.

As you can see from the screenshot, it contains the same information that is contained in the headers, a little easier for somebody to see. So flipping the option ServerTokens back to Prod and all you display is that the server is running apache. Flipping the ServerSignature option to Off does exactly that, there is no longer a signature at the bottom of the pages generated. The other option is Email. This is the same as On but adds a mailto: link where the server name is too. This may be handy if you want to allow people to track down the administrator of the server.

As I mentioned at the start of the post, this to me, falls into the ‘security through obscurity’ category, and should be one step towards making your server a little more secure, but definitely not the last. The less the person knows about a system, the more they’ll have to work to find out ways in.

After today’s little hiccup, I noticed the random image in the top right of my page throwing an error message, something about ERROR_STORAGE_FAILURE. A little digging about, I stumbled across an error in the daemon.log file…

Fortunately, most of the times, this is relatively easy to resolve.

As you can see from the Msg_text column, the command completed OK. A quick refresh of the blog page, and the error is now gone, and the random picture is back…

Running on from today’s outage, I’ve put a few more safety measures in place. I’ve always had old faithful Nagios watching over my servers, however sometimes a little more is needed. Introducing Monit into the picture…

Monit is a free open source utility for managing and monitoring, processes, files, directories and filesystems on a UNIX system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations.

Monit basically watches your processes, and takes actions based on criteria specified. The configurations are practically English, so bashing something together quickly to monitor the major culprits isn’t hard. Debian is even easier for general installs.

Then you just need to edit the config. The base config has an awful log of commenting, so it’s easy to understand, but here is an example of monitoring apache…

Without any knowledge of the Monit configuration system, I bet 90% of all admins can figure out what all of this means.

Once started, Monit will come to life every X minutes (based on configuration variable), check the process, make sure it’s running, and not exceeding the defined limits, and take actions if it is.

Stay tuned for more server modifications for better handling my customers.

As a follow up to the previous Tinkering with Temperature post, here is more on what I did, and how.

I made some alterations to my original schematic, as I was originally going with a serial to RJ45 conversion.  I figured I’d save myself some headaches, and use one of the Cisco console cables I had laying around.  This had a slightly different pinout on the RJ45 end, compared to the serial cable I was going to make.  Pinouts for the cable can be found all over the internet, including Cisco’s own site, but I used this as a reference.

Now I had my new schematic, I put it onto a board.  As with my schematic, KiCad helped out here too.  It mocked out the basic components, I just had to rearrange them, and then draw tracks.  I did read somewhere that there is an auto-route feature, but I couldn’t find/figure it out, not that mine was complicated, and I couldn’t figure out a few lines.  So I ultimately ended up with the image to the right.

Once drawn out, a simple case of printing on special paper, or you can do it the cheaper way of using magazine pages.  I went with the later.  When printed,  it’s just a case of taping the paper to the board, and applying heat.

When applying heat, you must remember to turn off the steam options, otherwise it won’t work properly.  This can take some time, depending on the toner, paper thickness, and the iron involved.  Once you’re satisfied with enough transfer, time to get the paper wet.  This will dissolve the paper, but leave the toner on the board.

Now is a good time to inspect the board, and make sure that any missing traces are covered.  Usually you can get away with a sharpie, but you can get special etch-resistant pens.  I went over all the lines again just to make extra sure.

At this point, we could etch the board, but it’s still a little large, so I trimmed off the larger side, and left the shortest excess attached.  This gave me a good point to hold, as well as a good indication of how well it was going.

Now it’s etching time.  Following the instructions carefully, I did this outside in a well ventilated area, with poor light.  Okay maybe I cannot follow all instructions well, but I did wear gloves, and did this away from anything metal.

After about 15 minutes of agitation, and careful inspection, the board was etched.  Note that part of the board wasn’t etched in the picture, but that was part that was in the excess area.  Another quick trim to remove that excess piece, so it’d fit nicely into the housing.

I borrowed Jeremy’s drill, and a bit from Rick. I quickly drilled the parts, and got to work soldering.  One thing to note is that when soldering, and constructing, follow your diagrams closely.  I originally soldered one of the diodes in backwards, so it wasn’t working until I desoldered it, and put it back in correctly.

I’d carefully designed the size of the PCB to be oversized, I could have clearly chopped an extra inch or so off, because it was to fit inside an old Linksys workgroup switch box.  This ended up being about 4×6.  The RJ45 connectors were a little low profile for the case, but it doesn’t matter too much, as it’s going to be hiding in the server room away from careful inspection of most people.

The final product looks like this.  Sits nicely inside the chassis, and looks pretty good.  It was a 5 port workgroup switch, with port 5 being disabled if you had an “uplink” cable in place.  In my case, port 6 is going to be the serial port.

Now for Monday, build some cables, once I’ve figured out all the lengths I need.  I did do a quick breadboard test with a RJ45 connector, and 2 wires crimped down to a breadboard to ensure it was working, and got good readings (compared with my fluke tester). 

I’ll power additional pictures once I have all the cables in place, and tucked into the racks.

I’ve been meaning to get some proper stats setup for my mail server, but until then, I’ve been mostly content with reading the LogWatch reports from my server. It has been dutifully analyzing my log files, and giving me all kinds of useful data, such as the mail stats from Saturday, 18th July…

The first interesting stat is for my tiny little mail server, I received 27,668 emails. Of that 27,242 of them were dropped at the door. 20k of which were RBLs. Of the handful that were left behind, half of them were marked as spam by Amavis and SpamAssassin.

Here are the full details…

 --------------------- Postfix Begin ------------------------ 

        4   SASL authentication failed 
 
   48.003M  Bytes accepted                        50,334,433
   77.976M  Bytes sent via SMTP                   81,763,280
    8.110M  Bytes delivered                        8,503,744
 ========   ================================================
 
      426   Accepted                                   1.54%
    27242   Rejected                                  98.46%
 --------   ------------------------------------------------
    27668   Total                                    100.00%
 ========   ================================================
 
      504   5xx Reject relay denied                    1.85%
       42   5xx Reject HELO/EHLO                       0.15%
     4239   5xx Reject unknown user                   15.56%
     1914   5xx Reject recipient address               7.03%
    20543   5xx Reject RBL                            75.41%
 --------   ------------------------------------------------
    27242   Total 5xx Rejects                        100.00%
 ========   ================================================
 
     1097   4xx Reject recipient address             100.00%
 --------   ------------------------------------------------
     1097   Total 4xx Rejects                        100.00%
 ========   ================================================
 
    22593   Connections made      
    12936   Connections lost (inbound) 
    22592   Disconnections        
      422   Removed from queue    
      115   Delivered             
      333   Sent via SMTP         
        4   Resent                
     7536   Policyd-weight        
 
        2   Connection failure (outbound) 
      184   Timeout (inbound)     
       28   Illegal address syntax in SMTP command 
        4   Numeric hostname      
       14   SMTP dialog error     
       28   Excessive errors in SMTP dialog 
     3484   Hostname verification errors 
        4   Hostname validation errors 
        7   SASL authenticated messages 
 
 
 
 ---------------------- Postfix End ------------------------- 

This reminds me of a need to setup more monitoring and statistics. For example, per domain logs for the domain owners to review (and me to find out who uses my system the most).

I have some interesting ideas for my servers, just got to get around to doing them…

I’m currently working on rebuilding a server on a project, and went to hit the tab key to auto-complete, when I was presented a delightful error…

$ vim RE-sh: <( compgen -d — ‘RE’ ): No such file or directory
-sh: <( eval compgen -f -X ‘*.@(o|so|so.!(conf)|a|rpm|gif|GIF|jp?(e)g|JP?(E)G|mp3|MP3|mp?(e)g|MPG|avi|AVI|asf|ASF|ogg|OGG|class|CLASS)’ — $(quote_readline $cur) ): No such file or directory

This is actually a reported bug in Debian [#502804] and is caused by using /bin/sh as a shell, rather than /bin/bash due to some POSIX compliant code.  A simple change to the shell until they release the fix..

usermod –s /bin/bash jangliss

And we’re all set again.

Today I’ve decided to finally shut down a trusty server.  It’s been the office firewall for 4+ years, happily chugging away on an old HP desktop, running Gentoo linux.

$ uptime
08:44:48 up 267 days, 17:51,  1 user,  load average: 0.00, 0.00, 0.00

It’s currently been running for nearly a year.  It’d be double that if we’d not had an issue with a snake, and the office transformed.  Before that, it’d been running without issues for nearly 2 years, where the failure before that was caused by a PSU fan failure.

It was recently phased out with some network reconfigurations, and is no longer needed…

Good night lestats.

I love one-liners to make my life easier.  Today I decided to clean up a directory full of CSV files, and compress them down.  After all, a 2MB CSV file can be reduced quite substantially, and I had 300 of them.

Here it is, quite simply loops through the directory of CSVs, and zips the file up, keeping the name the same.

for i in `ls -1 *.csv`; do zip -m ${i%csv}zip ${i}; done

This all goes on a single line, but can easily be written out over multiple lines, like this:

for i in `ls -1 *.csv`;

do

  zip -m ${i%csv}zip ${i}

done

Doing this reduced the folder size down from 600MB, to about 70MB, about an eighth the size.

I’ve always been in favor of removing the standard Sendmail install on most Linux boxes, in favor of Postfix. I’ve found it easier to use, and easier to configure. Having a simple configuration file that doesn’t need compiling is always helpful too. However, in some cases, you really don’t like to touch the delicate balance of an ancient server, in fear of it falling over in a spectacular death.

Due to the nature of a lot of viruses, our firewall blocks outbound SMTP, with the exception of a few hosts. This allows for services to use the server as a mail relay, whilst not allowing infected clients to send bad emails. Due to this, I had to figure out how to reconfigure sendmail on one of our servers to use our postfix based server as a mail relay. This actually turns out to be relatively easy, a single line in the /etc/mail/sendmail.mc file. The bit that caught me was getting it to work.

RedHat is nice, in that they supply a make command in the /etc/mail directory to build new configurations. However, for some reason, the default install of sendmail is missing some critical files that allow the rebuild of the configs to work. The first step to setting up the relaying was to add the line:

DEFINE(`SMART_HOST',`my.relay.host')

This was added into the /etc/mail/sendmail.mc file. Then the idea is you’re supposed to be able to execute:

make -C /etc/mail

This spewed out some data about it processing the directories. I then recycled sendmail. I gave the new setup a quick test, and found it was still trying to direct send. I verified the files, and it was configured right. However, I noticed something odd. Using ls -lt I found the sendmail.cf file (the compiled configuration file) had not been updated. This was odd, as it should have built it.

Having had experience with modifying the sendmail configurations before, I knew the make command was simply issuing an m4 command to compile the configurations. On a hunch, I tried running the command myself:

m4 sendmail.mc > sendmail.cf

This is where the hint of the issue came in…

Cannot open /usr/share/sendmail-cf/m4/cf.m4

So I was missing files required to build the new configuration files. A quick google search showed I needed to have the sendmail-cf package installed. As RedHat 9 is pretty old, I didn’t expect the RedHat FTP servers to still hold the files, so I hit RPMFind, and searched for sendmail-cf. A handful of results returned, but I found the one specifically for RedHat9.

After downloading the RPM file, and installing it using rpm -Uvh file I was then able to execute the make command as I had done before, but this time with a little more success (the files updated). Now it was time to restart sendmail again… This wasn’t as successful….

sendmail[24223]: NOQUEUE: SYSERR(root): No local mailer defined

sendmail[24223]: NOQUEUE: SYSERR(root): QueueDirectory (Q) option must be set

That’s not right. So a quick look in the sendmail.mc file again, I found that the local delivery option was not set, so I added the following line:

MAILER(local)

Then rebuilt the configurations using the make command again, and restarted Sendmail. This time with some success. I then tested the connection, ensuring that it relayed through the remote server…

sendmail[24468]: l7FFWqWF024468: from=jon@netdork, size=45, class=0, nrcpts=1, msgid=<20070815

1533.l7FFWqWF024468@origsmtp_server>, proto=SMTP, daemon=MTA, relay=localhost.localdomain [127.0.0.1]

sendmail[24472]: l7FFWqWF024468: to=jon@netdork.net, delay=00:00:13, xdelay=00:00:01, mailer=relay, pri=300

15, relay=my.relay.host [172.16.10.5], dsn=2.0.0, stat=Sent (Ok: queued as A979D6BCB4)

Now the server is behaving as I need it to.

Technorati Tags: Work, Tech, Linux, Sendmail, Postfix