Matt Simmons posted a question today on Twitter…

The cool thing about PowerShell, like Perl, if it can be done one way, there are probably 10 other ways to achieve the same thing. Here are a couple…

Method 1 – ADSI

This method will show the full DN of each group the user is a member of.

Method 2 – Quest’s Active Directory PowerPack

I’ve been a PowerGUI user for a bit, and the Quest AD PowerPack is pretty damn cool. You don’t have to buy the full product, there is an open source version of PowerGui, but the AD tools are at Quest’s site here. This one is a little easier, as you don’t have to know the full DN for the user, you can search by name, phone, all kinds. Here is a simple example:

This too returns the full DN for the groups.

Method 3 – Windows 2008 R2 AD Modules

Now there are some requirements behind this one, so go check them out, the least of which is a Windows 2008 R2 server. It’s a little unclear if the R2 box has to be a domain controller or not, but the 2008R2 Active Directory Web Services (ADWS) must be installed on at least one DC.

I cannot validate the syntax exactly on this one, as I don’t have the facilities available to do so.

But there you have it, 3 quick commands to get you access to the user’s groups.

EDIT: I found another one, which uses the ADSI method, but calls up the Directory Services modules…

This one requires a little understanding of LDAP searches, but not too bad otherwise.

This one has been sitting idle for a couple of weeks.  Lethargy kicked in, and work was consuming too much of my brain power, but here it is, I’m working on getting it out the door.  In this episode, we’ll be running through the basics of svn, and getting it working with the files we have in place, adding new files, reverting, and planning for big upgrades… Note, this is more of an SVN primer than anything.  If you know how to tag, branch, revert, have a good general knowledge of SVN, this one can be skipped.

Modifying a File

The machine I’m working on has Bind installed.  First lets edit the named.conf.local file, and add reference to a new zone file.

cat >> /etc/bind/named.conf.local

zone "local.netdork.net" {
    type master;
    file "/etc/bind/db.local.netdork.net";
};

Hit CTRL-D and it’ll quit the cat command, and append the data to the file (note I used >>).  We can verify this by simply doing a quick cat again:

Now we see a handful of lines that look like this:

The –m argument allows you to add a message.  If you need to add a longer message, such as a multi-line entry, omit that argument, and the text, and SVN should prompt in your default editor.

Adding a New File

Now we’ve added our reference, lets create the zone file quickly. I’m going to cheat in this case for speed, and copy Debian’s db.local file to the new name.

Reverting a change

You know we’ve all done it.  Edited a file, it didn’t work out, but you forgot what you changed. That’s what version control does, keeps previous versions, saves our bacon when we need it. We’re going to go extreme here, we’re going to trash the main Bind configuration with some random data. Ignoring the syntax if you don’t know what it does, I’m basically taking the random entropy source, and dumping it into the named.conf file, not a good thing, that’s for sure.

And we’re back to how we were.  That’s great, it means Bind will be happy again.

What happens if it’s a change we committed? What if we made a change, committed it, and discovered it was an issue a week later, but we don’t remember exactly what it was?  First, lets look at the log, and see when the file was changed, revision, and the (hopefully) useful log entry we added…

There, we see there were 2 changes committed to this file. Part of the initial commit (yes I created a bad log entry there), and r7, which is the one we just made. So lets say we want to get back to r5, unfortunately svn doesn’t have a “rollback” to a specific version, you have to merge backwards the changes.  This isn’t too difficult to do, we’ll use the example from the modifying files at the top to roll back…

In this example, we looked at the log, and see the entry was r7 for when we committed the entry for the local.netdork.net zone entry, and r5 was the last change before that.  So the command merges from 7 down to 5. svn status will now show that the file has changed, and diff will show the inverse of the top second.  Commit just solidifies the work.

Handling Major Upgrades

Major upgrades are difficult, we want a point in time restore.  What happens if the upgrade goes badly, and we need to roll back a couple of packages, but the config files don’t get reverted? Whilst SVN will handle that using the above methods, we’re going to want to keep a way of finding those changes easily.  This is where tags come in handy.  They’re a complete copy of an existing branch. We’re going to make a folder for tags, and make a full copy of the current config.

You’ll see the standard SVN commit confirmation, now we can checkout the results using a command we used in part II.

If you skim through the results, you’ll now see a tags folder, with a subfolder 20100809deb505, and a complete copy of Trunk.

Now we’ve done that, and we do the upgrade, we can go back at any point, and see what was changed…

Finishing up

There is one thing I didn’t do above, but thought I’d point it out at the bottom to emphasis it a little bit.  Some files need special permissions, and we want to retain those too.  You might remember in the previous edition, we introduced asvn, which was a wrapper script around the svn command.  Once you’ve added your files to svn, use the asvn script to update svn with the info flags for the file, and check that in too.  You can see how to use the command in part II.

Now we have a good basis for storing and archiving our configurations, next time we’ll look at monitoring them for changes that aren’t checked in.

Lori MacVittie has an excellent article on passive host monitoring in an F5 Big-IP environment.  Introduced in v10, BIG-IP load balancers now support something called Inband Monitoring.  This basically sniffs the traffic, and responds based on the rule you define.  Lori put an example up which looks for the HTTP error codes above 500, and after 3 errors seen in the responses, it marks that host as unavailable.  This is a great feature as it doesn’t add any load to the web servers with active checks, and if you have ever looked at a host that i s behind a load balancer, log analysis can be a pain.  The cool thing is, after the host has been disabled, the passive check can enable your regular active checks to bring the host back up when the issue is resolved. Some excellent stuff.

Matt Simmons, of Standalone Sysadmin fame, has a great post up about SNMP, giving a good all around introduction to SNMP.  I highly recommend reading it if you’re getting started in networks, or even if you just want to brush up.

In my previous post about PowerShell and BITS, I stumbled on a weird quirk in the ForEach-Object function, which had me scratching my head for a bit.

In most programming languages, where you have a for, loop, while, foreach, or other such loop, there is often a set of control functions that go with it. Continue and Break. These functions alter the way a loop is behaving at that time. A call to Continue will stop the current loop iteration at that point in the code, and jump onto the next. A simple example would look like this:

This code simply takes the variable $i, sets it to 0, and whilst $i is less than 5, it loops through the code, each iteration increasing the value of $i by 1. The if statement executes special code when $i gets to the number 2. Now if we run this code, we get the following:

What happened was the counter hit 2, and the code said that it was done with this iteration, and continue processing the outer loop. A Break on the other hand just tells the loop it’s done processing, and to stop looping, and get out. Using the same example as above, but changing Continue to Break we get the output that looks like this:

This is because PowerShell has been told to stop processing any further.

Now for the quirk. I discovered that ForEach-Object isn’t actually a programmatic keyword, it’s a cmdlet. How does this affect the usage of the control functions? Lets look at my code from my BITS article, and compare it to what I had.

As you can see, I’m using the continue function inside the array of strings for the images. If the string value is empty, I skip onto the next one. This is what the original code looked like:

Whilst the code seems pretty similar, there is one distinct difference. The first block of code is inside a loop, the second set is not. Even though you think that ForEach-Object seems like a loop, it’s a cmdlet that does not behave the same way that a loop does. How does this affect the behavior of the code? Badly! Because there is no loop, the control function looks for the next operation it can operate on. It turns out that the next function it can operate on is the entire script. It essentially kills the script right where the line is. This means no processing of other objects in the ‘loop’, no continuing with the code further in the script, no nice handling of anything else in the script. It’s done, finished, over. My hint something wasn’t behaving right was the fact I should have had several thousand images, and yet I only had about 150. It took 4 or 5 attempts, as well as various debug statements throughout the code to figure out why this was happening. After I realized where it was going wrong, a Google search dropped me over to James Manning’s post PowerShell gotcha – foreach keyword vs. foreach-object cmdlet.

So if you ever find yourself in need of an object loop, remember that ForEach-Object isn’t really a keyword, it’s a cmdlet, and does not behave the same.

See any mistakes? Want to add your feedback? Leave me a note in the comments, I love to hear from you.

Ever had to download a bunch of files from a website, and didn’t want to have to write an HTTP handler for it? This comes up on a regular basis here. We get requests from customers that have transferred from one vendor to another, and want to import all their photos to us. Here is how I solve it.

What is BITS?

BITS, or Background Intelligent Transfer Service, is a service bundled with Windows that is used to transfer files from websites. It was originally introduced in Windows XP RTM back in October 2001. You may not know it, but your computer uses it all the time if you have Windows Automatic Updates enabled. It is an intelligent service that asynchronously transfers files, automatically adjusting bandwidth usage, throttling downloads, as bandwidth becomes available. It can even be used to download files across computer starts.

How is it useful in PowerShell?

BITS is really easy to use, a few commands which I’ll show shortly, and you can download a bunch of files quickly, without having to write HTTP handler. In my case above, I receive text files, with a list of image URLs, that need to be imported.

Here is how to do a very simple, single file, transfer.

Relatively simple, a few lines of code for a single file. However, BITS really shines when you’re working with multiple files to download.

As the files I receive are quite often comma-separated values (CSV) files, we can use the rather handy Import-CSV function, and we build up from there.

This is a little more complicated, but is still pretty simple. It introduces Add-BitsFile which allows you to add files to an existing job, and a simple ForEach loop to go through an array of URLs.

I’ve made a few assumptions here, and thrown out all kinds of error handling in favor of a quick code turn around. For example, the job state of a BITS job can be one of nine options (which you can see here), but I assume only two. I’m also not handling cases where I exceed the number of jobs, but that never happens because I only ever have one job running at a time. I’ll probably work on tidying it up a little for other people’s use, but for now, it’s just me using it where I work, so the impact of an error is very minimal.

Give it a shot, download some images off of Flickr¹, or go a little further, and test it against downloads from Microsoft.

Got any questions? Hints? Tips? Leave them in the comments, I’d love to hear from you.

1. if they are yours, or have Creative Commons license

Whilst driving back from our 4th of July celebrations in San Antonio, I got alerted to a server running low on disk space.  As it hadn’t hit critical, and there was still a fair bit of space left, I decided to wait until I got home.  When I looked at the server, I pulled up my trusted space analyzer, TreeSize.

A quick analysis of the drive, and a report showed that there was a hefty chunk of space lost to the WinSXS directory¹, and 8GB of Files in the root of C:.  This was a little odd, I know I usually use c:\temp from time to time, but never the root.  To find out what was there, I first had to show hidden and system files.  To do this, go to Tools, then Folder Options, then onto the View tab.  There are 3 options there that we need to change.

1. Change “Hidden files and folders” to “Show hidden files and folders”
2. Uncheck “Hide extensions for known file types”
3. Uncheck “Hide protected operating system files (Recommended)”

I highly recommend doing 2 all the time, you never know when somebody might sent a .txt.exe file².  Changing 3 will prompt that it isn’t recommended, but do it anyway, the files I was having issues with were still hidden after changing the first two.

After making these changes, two files appeared that were the culprits to the lost disk space.  Pagefile.sys and Hiberfil.sys.  I’ll leave the explanation of the Pagefile to the experts.  Hiberfil.sys is a file that is used to handle the state of memory, and running applications when you hibernate your computer. 

What is hibernation?

Hibernation is a heavy sleep state for computers where all the running processes are frozen, the memory is locked, and all the states are saved into that file, then the computer is powered down.  When the computer is turned back on, the operating system sees this state file, and starts to reload everything from there.  This makes it appear like the computer was never even turned off.  The applications are resumed, everything is back to the state it was before hibernation was kicked off.

Why is that bad?

Generally speaking, it isn’t.  In fact, it’s a great thing for home computers.  When you’re done reading your email at night, if you hibernate your computer, and hop into bed, your computer will be off, saving power.  However this isn’t so good for servers.  In 95% of cases, servers don’t ever need to hibernate.  They usually have tasks, and operations that they need to run all the time, which means putting them in a state of hibernation is a bad thing as the tasks won’t be run.

How do you stop this?

Usually, servers have this feature turned off, but for some reason this particular one had it enabled.  I’ve done some looking around, and it seems it’s may have actually been a default option to have it enabled, but for some (lucky) reason a handful of our other servers don’t have it on.  In earlier versions of Windows, disabling the hibernation option was a case of removing a check mark from a box.  This was done in the Control Panel under Power Options, and the Hibernate tab.  However, in Windows 2008 this option was removed, and you now have to use a command line to manage this advanced option.

Within about 10 seconds of issuing that command, the Hiberfil.sys file disappeared, and I recovered 4GB of disk space.  You can read more about the powercfg command here.

1. If you’ve not read about this, I strongly recommend you do when planning out a Windows 2008 server build
2. A very old trick from virus/malware creators

In part II of my walk through on configuration management, I’m going to work on setting up Subversion (SVN). SVN is a version control system and, just like any version control system, is used to keep track of changes. The most common use is in a development environment with teams of developers so that everybody can work on the code, without having to worry about other people’s work. It’s obviously not limited there, it can be used as a form of backup, keeping a track of what you’ve changed, and when. This is how we’ll be using it for our configuration management.

Installing SVN

I’m going to make the assumption you know how to install Linux. If not, there are plenty of guides on doing so around the internet, and most come with a pretty easy to use installer. With that said, I am starting this off with a completely blank slate. I installed Debian 5.0.4 inside a VirtualBox instance. I’ll be installing SVN on the same server for now, but it could be installed anywhere that supports it.

The above command needs to be run as root. You can do the same as a normal user as long as you prefix the command with sudo.

Creating the repository

Next step is to create the repository we’ll be working with for the configurations.

You can name the repository whatever you want, I used thebe_config, Thebe is the name of the server, and the second part is obvious. Next we’ll create a group for those that will be working on the SVN project, and add the users. As it’s only me, that’s easy…

That’s the basic setup done… Now onto more fun stuff.

Using the repository

Generally speaking, SVN requires you to import a project, and checkout in another location. For example, a basic initial import, and check out would look something like this:

If you tried to checkout the project on-top of the existing code you already put up, you’d get an error like this:

This isn’t so good for us, because we can hardly delete /etc. That being said, SVN has it covered, with what they document as an “in-place ‘import’“. The really cool thing about this, they document exactly what we’re trying to do, version-control the /etc structure. So instead of importing /etc, we fake it.

Whilst it’s not documented in the FAQ there, the –parents option is similar to that of the normal mkdir, it recursively creates the directory structure down to the child we’re making. In this case, trunk/etc. After the last 2 commands, you should see a stream of data zip by you. In the above example, I added the entire content of /etc, you can restrict it to certain directories if you want, say apache2, or samba, or you can go the whole way, and do the above.

Now it’s all checked in, lets see what it looks like:

After that, you should see a nice structured view of your checked in directory structure.

A gotcha with SVN

Before we go messing with these files, and testing everything is working well, we need to note something about SVN, that is important to the /etc directory structure. File permissions. SVN does not record file permissions. This is an issue for some parts of the /etc directory, such as the shadow file, or the group file, or that htpasswd file you created to restrict users access.

SVN allows you to store properties information along side each file, this file can contain all kinds of information, such as copyright data, license information, and all that fun stuff¹. A crafty contributor too advantage of that, and created a script called asvn. This script is a wrapper around the usual svn command, and executes some work either before a commit, or after a checkout. So we need to grab a copy of this script:

After the last command, you’ll see a bunch of stuff zoom past, and you will be prompted for a commit message, I did “permissions fixup”, and hit CTRL X and yes. More stuff zooms by, and you should see a message appear that tells you the revision number has incremented. Now to see if it worked…

Excellent, now the last test, see what the checkout looks like, and see if it matches…

More scrolling of lots of boring text, but you should notice stuff like “checking {somefile} for symlinks”. This is the asvn script doing it’s work, and rebuilding permissions and symlinks. In my example, I’d checked in everything, which includes the precious “shadow” file (I’d not recommend adding that one, add it to the svn:ignore list). I did so to provide a sample file with differing permissions. The original file is locked down to specific user and group:

Lines starting with the $ are the commands I executed. The rest is the response from the server. As you can see, the permissions match.². Now the results in our test checkout…

Permissions match, size matches, so it looks like SVN now has a good copy of the file properties, and the version I have checked in is the same as the one I have in /etc. As a side note, it should be worth remembering that some of the permissions cannot be set by anybody but root, so a lot of this work will probably require root, or sudo access. That being said, you’re editing /etc, so you should already be doing that.

Wrap-up

So far, we’ve covered why we want to use a version control repository for our configurations. We’ve also setup SVN, and got the basics going, including extended SVN to include extra data to handle file properties. Next up is the basics of using SVN, and how it’ll be helping us out here in /etc and configuration land.

If you see any errors, or have any suggestions, feel free to leave them in the comments. Want me to write about something else? Drop me an idea, and I’ll see what I can do.

1. You can read more on SVN properties here
2. Not sure how the -rw-r—– matches 640? I highly recommend checking out the excellent StandAlone SysAdmin’s great post on numeric unix permissions

I’ve been asked to write a post on configuration management, and version control by a friend, Steven Klassen. Instead of a single post, I’m going to break this into several posts as some parts might be unimportant to some, and they can easily skip a whole post. I’ll be posting this over a few days, so stay tuned. If you’re not subscribed to my RSS feed, now would be a great time.

The Request

Earlier today, Steven asked if I used any form of version control. It was a leading question, straight into “dev only or system files”? His questions were targeted, he was searching for guidance¹. Another question or two later, and it boils down to this:

• Do you use version control for system files?
• Files like domain zone files?
• What about multi-user environments?
• Can you write it up?

My take away on the questions boils down to two things; configuration management, and backups. Anybody that has managed any type of system has, at some point, messed up the configuration file, and forgotten what they did to get there in the first place. Those who have been at it a while will usually do a quick copy before they do any modifications. But, as Steven found out in the past, best laid plans can be foiled by a simple typo.

Version Control

This is what version control is all about. It acts as a backup, whilst keeping a running history of what you did between changes. Version control, or revision control as it’s also known, is big business. Anywhere you find a big name development company, you’ll find they’ve probably developed a revision control system to go with whatever they’re offering. Take a look at this Wikipedia entry comparing just a handful of them.

Configuration Management and Version Control

You don’t have to be a genius to figure out how useful version control can be for configuration management, but in case your brain often goes off to lala land like mine, here is a run down on a few reasons you should really consider using it…

• What did I just change again? And why is that service not starting any more? Oh hell…
• 6 months down the road, can you figure out why you changed that line to Apache to start 10 daemons instead of 5? No? Most version control systems allow you to add comments to your commits.
• As above, but why did Fred in your team change that entry back down to 7 a week later?
• In the words of Homer… Doh!! I wasn’t supposed to delete/overwrite that file!

So, there are 4 obvious cases as to why you’d want to use version control.

The Breakdown

As I said at the beginning, I’ll be breaking this down into several parts. I expect them to be something along these lines:

• Introduction
• Setting up SVN and the base repository
• Using your repository
• Educating your team members
• Maintenance and monitoring
• Conclusion, and follow-up ideas.

If you’re interested, keep an eye open for the rest of the series. If not, send me something you’d like to see me write about. I’m open to suggestions, and will probably butcher anything you can throw at me.

1. At least that’s how I perceived it

The Nubby Admin has a great post on uptimes, and the old fascination of having a large uptime¹. Okay, you can get your minds out the gutter now, not that kind of up time. We’re talking servers here.

The post covers a hidden fear, and the goods and bads of large server uptimes. A good read, and one you should look at if you’re watching your server rolling over into the third year of being running. I get a mention (well more of a quote), and seem to fall in with the general crowd, large server uptimes are generally bad.

Go read, enjoy, learn something new from the great minds Wesley is surrounding himself with (myself excluded).

1. Netcraft even has a page dedicated to it