A common habit in a lot of IT shops is computer cloning. This saves time, and saves on mistakes. If you have a good image of one computer, and you use it to duplicate to another, you know the other machine is going to be good. You get to apply all your corporate policies, security software, anti-virus, group policies, etc etc, all in one single step. With this in mind, this is what we did with some SQL clusters we built 6 months ago.

Whilst posting my latest post on a SQL server issue, I realized that my blog is 5 years old today. A long and random path, with long spates of not posting. I think I’ve stepped up a little more recently, but I could probably do more. So what has 5 years got me?

Back in June, Microsoft released a Sharepoint security update (MS10-039) which reportedly resolved an issue which allowed security privileges to be increased. At the time, I hadn’t deployed the update to our TFS server because we were in the midst of some big project work, and I didn’t want to disrupt the server. Last week, I finally got around to playing patch catchup, and deployed this patch. It didn’t go as smoothly as one would hope…

Matt Simmons posted a question today on Twitter…

Anyone know how to show a user’s group membership in powershell? — Matt Simmons

The cool thing about PowerShell, like Perl, if it can be done one way, there are probably 10 other ways to achieve the same thing. Here are a couple…

This one has been sitting idle for a couple of weeks. Lethargy kicked in, and work was consuming too much of my brain power, but here it is, I’m working on getting it out the door. In this episode, we’ll be running through the basics of svn, and getting it working with the files we have in place, adding new files, reverting, and planning for big upgrades… Note, this is more of an SVN primer than anything. If you know how to tag, branch, revert, have a good general knowledge of SVN, this one can be skipped.

Modifying a File

The machine I’m working on has Bind installed. First lets edit the named.conf.local file, and add reference to a new zone file.

cat >> /etc/bind/named.conf.local

zone "local.netdork.net" {
    type master;
    file "/etc/bind/db.local.netdork.net";
};

Hit CTRL-D and it’ll quit the cat command, and append the data to the file (note I used »). We can verify this by simply doing a quick cat again:

cat /etc/bind/named.conf.local

Now we’ve verified we have content added to the file, lets see what SVN thinks about our new directory changes.

cd /etc
svn diff

Now we see a handful of lines that look like this:

Index: bind/named.conf.local
===================================================================
— bind/named.conf.local       (revision 5)
+++ bind/named.conf.local       (working copy)
@@ -6,3 +6,9 @@
// organization
//include "/etc/bind/zones.rfc1918";
 
+
+zone "local.netdork.net" {
+    type master;
+    file "/etc/bind/db.local.netdork.net";
+};
+

This can be sent to a file, and used as a patch on another system. The format is a predefined layout that tells another system all the information it needs to update another file. For example, lines starting with a – are removed from a file, those with a + are added to the file. This shows that the difference between my SVN backup, and my live copy is the changes I’d just made. Now as I want to keep those, I’m going to check them in.

svn commit -m "Adding local.netdork.net zone reference" named.conf.local

The –m argument allows you to add a message. If you need to add a longer message, such as a multi-line entry, omit that argument, and the text, and SVN should prompt in your default editor. Adding a New File

Now we’ve added our reference, lets create the zone file quickly. I’m going to cheat in this case for speed, and copy Debian’s db.local file to the new name.

cp /etc/bind/db.local /etc/bind/db.local.netdork.net

Now bind is going to be happy, however, we now have a new file to add to svn. We can see this by using the status command.

svn status
? bind/db.local.netdork.net
svn add bind/db.local.netdork.net
svn commit -m "Adding zone file for local.netdork.net for Bind" bind/db.local.netdork.net

The ? shows the new file, modified files will show M, deleted D, and A will show added files with no commits.

Reverting a change

You know we’ve all done it. Edited a file, it didn’t work out, but you forgot what you changed. That’s what version control does, keeps previous versions, saves our bacon when we need it. We’re going to go extreme here, we’re going to trash the main Bind configuration with some random data. Ignoring the syntax if you don’t know what it does, I’m basically taking the random entropy source, and dumping it into the named.conf file, not a good thing, that’s for sure.

 dd bs=1024 count=1 if=/dev/urandom of=/etc/bind/named.conf
  1+0 records in
  1+0 records out
  1024 bytes (1.0 kB) copied, 0.000929448 s, 1.1 MB/s
head /etc/bind/named.conf
  #lots of random stuff#
svn status
M    bind/named.conf

So using my above, rather destructive code, I’ve overwritten /etc/named.conf with 1MB of random data. That’s okay, we’re going to call on the revert command.

svn revert bind/named.conf
Reverted ‘bind/named.conf’
head bind/named.conf

And we’re back to how we were. That’s great, it means Bind will be happy again.

What happens if it’s a change we committed? What if we made a change, committed it, and discovered it was an issue a week later, but we don’t remember exactly what it was? First, lets look at the log, and see when the file was changed, revision, and the (hopefully) useful log entry we added…

svn log bind/named.conf.local
------------------------
r7 | root | 2010-08-09 21:38:36 -0500 (Mon, 09 Aug 2010) | 1 line

Adding local.netdork.net zone reference
------------------------
r5 | root | 2010-06-30 20:06:27 -0500 (Wed, 30 Jun 2010) | 2 lines

Updates
------------------------

There, we see there were 2 changes committed to this file. Part of the initial commit (yes I created a bad log entry there), and r7, which is the one we just made. So lets say we want to get back to r5, unfortunately svn doesn’t have a “rollback” to a specific version, you have to merge backwards the changes. This isn’t too difficult to do, we’ll use the example from the modifying files at the top to roll back…

cd /etc/bind
svn log named.conf.local
svn merge –revision 7:5 named.conf.local
svn status
svn diff named.conf.local
svn commit -m "Removing local.netdork.net from Bind" named.conf.local

In this example, we looked at the log, and see the entry was r7 for when we committed the entry for the local.netdork.net zone entry, and r5 was the last change before that. So the command merges from 7 down to 5. svn status will now show that the file has changed, and diff will show the inverse of the top second. Commit just solidifies the work.

Handling Major Upgrades

Major upgrades are difficult, we want a point in time restore. What happens if the upgrade goes badly, and we need to roll back a couple of packages, but the config files don’t get reverted? Whilst SVN will handle that using the above methods, we’re going to want to keep a way of finding those changes easily. This is where tags come in handy. They’re a complete copy of an existing branch. We’re going to make a folder for tags, and make a full copy of the current config.

svn mkdir –parents -m "Creating tags folder" file:///srv/svn/thebe_config/tags
svn copy -m "Backup for Thebe Debian 5.0.5 on 08/09/2010" \
  file:///srv/svn/thebe_config/trunk \
  file:///srv/svn/thebe_config/tags/20100809deb505

You’ll see the standard SVN commit confirmation, now we can checkout the results using a command we used in part II.

svnlook tree /srv/svn/thebe_config | more

If you skim through the results, you’ll now see a tags folder, with a subfolder 20100809deb505, and a complete copy of Trunk.

Now we’ve done that, and we do the upgrade, we can go back at any point, and see what was changed…

svn diff –old=file:///srv/svn/thebe_config/tags/20100809deb505 \
  –new=file:///srv/svn/thebe_config/trunk/

Finishing up

There is one thing I didn’t do above, but thought I’d point it out at the bottom to emphasis it a little bit. Some files need special permissions, and we want to retain those too. You might remember in the previous edition, we introduced asvn, which was a wrapper script around the svn command. Once you’ve added your files to svn, use the asvn script to update svn with the info flags for the file, and check that in too. You can see how to use the command in part II.

Now we have a good basis for storing and archiving our configurations, next time we’ll look at monitoring them for changes that aren’t checked in.

Lori MacVittie has an excellent article on passive host monitoring in an F5 Big-IP environment. Introduced in v10, BIG-IP load balancers now support something called Inband Monitoring. This basically sniffs the traffic, and responds based on the rule you define. Lori put an example up which looks for the HTTP error codes above 500, and after 3 errors seen in the responses, it marks that host as unavailable. This is a great feature as it doesn’t add any load to the web servers with active checks, and if you have ever looked at a host that i s behind a load balancer, log analysis can be a pain. The cool thing is, after the host has been disabled, the passive check can enable your regular active checks to bring the host back up when the issue is resolved. Some excellent stuff.

Matt Simmons, of Standalone Sysadmin fame, has a great post up about SNMP, giving a good all around introduction to SNMP. I highly recommend reading it if you’re getting started in networks, or even if you just want to brush up.

In my previous post about PowerShell and BITS, I stumbled on a weird quirk in the ForEach-Object function, which had me scratching my head for a bit.

Ever had to download a bunch of files from a website, and didn’t want to have to write an HTTP handler for it? This comes up on a regular basis here. We get requests from customers that have transferred from one vendor to another, and want to import all their photos to us. Here is how I solve it.

Whilst driving back from our 4th of July celebrations in San Antonio, I got alerted to a server running low on disk space. As it hadn’t hit critical, and there was still a fair bit of space left, I decided to wait until I got home. When I looked at the server, I pulled up my trusted space analyzer, TreeSize.