Matt Simmons, of Standalone Sysadmin fame, has a great post up about SNMP, giving a good all around introduction to SNMP.  I highly recommend reading it if you’re getting started in networks, or even if you just want to brush up.

In my previous post about PowerShell and BITS, I stumbled on a weird quirk in the ForEach-Object function, which had me scratching my head for a bit.

In most programming languages, where you have a for, loop, while, foreach, or other such loop, there is often a set of control functions that go with it. Continue and Break. These functions alter the way a loop is behaving at that time. A call to Continue will stop the current loop iteration at that point in the code, and jump onto the next. A simple example would look like this:

This code simply takes the variable $i, sets it to 0, and whilst $i is less than 5, it loops through the code, each iteration increasing the value of $i by 1. The if statement executes special code when $i gets to the number 2. Now if we run this code, we get the following:

What happened was the counter hit 2, and the code said that it was done with this iteration, and continue processing the outer loop. A Break on the other hand just tells the loop it’s done processing, and to stop looping, and get out. Using the same example as above, but changing Continue to Break we get the output that looks like this:

This is because PowerShell has been told to stop processing any further.

Now for the quirk. I discovered that ForEach-Object isn’t actually a programmatic keyword, it’s a cmdlet. How does this affect the usage of the control functions? Lets look at my code from my BITS article, and compare it to what I had.

As you can see, I’m using the continue function inside the array of strings for the images. If the string value is empty, I skip onto the next one. This is what the original code looked like:

Whilst the code seems pretty similar, there is one distinct difference. The first block of code is inside a loop, the second set is not. Even though you think that ForEach-Object seems like a loop, it’s a cmdlet that does not behave the same way that a loop does. How does this affect the behavior of the code? Badly! Because there is no loop, the control function looks for the next operation it can operate on. It turns out that the next function it can operate on is the entire script. It essentially kills the script right where the line is. This means no processing of other objects in the ‘loop’, no continuing with the code further in the script, no nice handling of anything else in the script. It’s done, finished, over. My hint something wasn’t behaving right was the fact I should have had several thousand images, and yet I only had about 150. It took 4 or 5 attempts, as well as various debug statements throughout the code to figure out why this was happening. After I realized where it was going wrong, a Google search dropped me over to James Manning’s post PowerShell gotcha – foreach keyword vs. foreach-object cmdlet.

So if you ever find yourself in need of an object loop, remember that ForEach-Object isn’t really a keyword, it’s a cmdlet, and does not behave the same.

See any mistakes? Want to add your feedback? Leave me a note in the comments, I love to hear from you.

Ever had to download a bunch of files from a website, and didn’t want to have to write an HTTP handler for it? This comes up on a regular basis here. We get requests from customers that have transferred from one vendor to another, and want to import all their photos to us. Here is how I solve it.

What is BITS?

BITS, or Background Intelligent Transfer Service, is a service bundled with Windows that is used to transfer files from websites. It was originally introduced in Windows XP RTM back in October 2001. You may not know it, but your computer uses it all the time if you have Windows Automatic Updates enabled. It is an intelligent service that asynchronously transfers files, automatically adjusting bandwidth usage, throttling downloads, as bandwidth becomes available. It can even be used to download files across computer starts.

How is it useful in PowerShell?

BITS is really easy to use, a few commands which I’ll show shortly, and you can download a bunch of files quickly, without having to write HTTP handler. In my case above, I receive text files, with a list of image URLs, that need to be imported.

Here is how to do a very simple, single file, transfer.

Relatively simple, a few lines of code for a single file. However, BITS really shines when you’re working with multiple files to download.

As the files I receive are quite often comma-separated values (CSV) files, we can use the rather handy Import-CSV function, and we build up from there.

This is a little more complicated, but is still pretty simple. It introduces Add-BitsFile which allows you to add files to an existing job, and a simple ForEach loop to go through an array of URLs.

I’ve made a few assumptions here, and thrown out all kinds of error handling in favor of a quick code turn around. For example, the job state of a BITS job can be one of nine options (which you can see here), but I assume only two. I’m also not handling cases where I exceed the number of jobs, but that never happens because I only ever have one job running at a time. I’ll probably work on tidying it up a little for other people’s use, but for now, it’s just me using it where I work, so the impact of an error is very minimal.

Give it a shot, download some images off of Flickr, or go a little further, and test it against downloads from Microsoft.

Got any questions? Hints? Tips? Leave them in the comments, I’d love to hear from you.

Whilst driving back from our 4th of July celebrations in San Antonio, I got alerted to a server running low on disk space.  As it hadn’t hit critical, and there was still a fair bit of space left, I decided to wait until I got home.  When I looked at the server, I pulled up my trusted space analyzer, TreeSize.

A quick analysis of the drive, and a report showed that there was a hefty chunk of space lost to the WinSXS directory, and 8GB of Files in the root of C:.  This was a little odd, I know I usually use c:\temp from time to time, but never the root.  To find out what was there, I first had to show hidden and system files.  To do this, go to Tools, then Folder Options, then onto the View tab.  There are 3 options there that we need to change.

1. Change “Hidden files and folders” to “Show hidden files and folders”
2. Uncheck “Hide extensions for known file types”
3. Uncheck “Hide protected operating system files (Recommended)”

I highly recommend doing 2 all the time, you never know when somebody might sent a .txt.exe file.  Changing 3 will prompt that it isn’t recommended, but do it anyway, the files I was having issues with were still hidden after changing the first two.

After making these changes, two files appeared that were the culprits to the lost disk space.  Pagefile.sys and Hiberfil.sys.  I’ll leave the explanation of the Pagefile to the experts.  Hiberfil.sys is a file that is used to handle the state of memory, and running applications when you hibernate your computer. 

What is hibernation?

Hibernation is a heavy sleep state for computers where all the running processes are frozen, the memory is locked, and all the states are saved into that file, then the computer is powered down.  When the computer is turned back on, the operating system sees this state file, and starts to reload everything from there.  This makes it appear like the computer was never even turned off.  The applications are resumed, everything is back to the state it was before hibernation was kicked off.

Why is that bad?

Generally speaking, it isn’t.  In fact, it’s a great thing for home computers.  When you’re done reading your email at night, if you hibernate your computer, and hop into bed, your computer will be off, saving power.  However this isn’t so good for servers.  In 95% of cases, servers don’t ever need to hibernate.  They usually have tasks, and operations that they need to run all the time, which means putting them in a state of hibernation is a bad thing as the tasks won’t be run.

How do you stop this?

Usually, servers have this feature turned off, but for some reason this particular one had it enabled.  I’ve done some looking around, and it seems it’s may have actually been a default option to have it enabled, but for some (lucky) reason a handful of our other servers don’t have it on.  In earlier versions of Windows, disabling the hibernation option was a case of removing a check mark from a box.  This was done in the Control Panel under Power Options, and the Hibernate tab.  However, in Windows 2008 this option was removed, and you now have to use a command line to manage this advanced option.

Within about 10 seconds of issuing that command, the Hiberfil.sys file disappeared, and I recovered 4GB of disk space.  You can read more about the powercfg command here.

In part II of my walk through on configuration management, I’m going to work on setting up Subversion (SVN). SVN is a version control system and, just like any version control system, is used to keep track of changes. The most common use is in a development environment with teams of developers so that everybody can work on the code, without having to worry about other people’s work. It’s obviously not limited there, it can be used as a form of backup, keeping a track of what you’ve changed, and when. This is how we’ll be using it for our configuration management.

Installing SVN

I’m going to make the assumption you know how to install Linux. If not, there are plenty of guides on doing so around the internet, and most come with a pretty easy to use installer. With that said, I am starting this off with a completely blank slate. I installed Debian 5.0.4 inside a VirtualBox instance. I’ll be installing SVN on the same server for now, but it could be installed anywhere that supports it.

The above command needs to be run as root. You can do the same as a normal user as long as you prefix the command with sudo.

Creating the repository

Next step is to create the repository we’ll be working with for the configurations.

You can name the repository whatever you want, I used thebe_config, Thebe is the name of the server, and the second part is obvious. Next we’ll create a group for those that will be working on the SVN project, and add the users. As it’s only me, that’s easy…

That’s the basic setup done… Now onto more fun stuff.

Using the repository

Generally speaking, SVN requires you to import a project, and checkout in another location. For example, a basic initial import, and check out would look something like this:

If you tried to checkout the project on-top of the existing code you already put up, you’d get an error like this:

This isn’t so good for us, because we can hardly delete /etc. That being said, SVN has it covered, with what they document as an “in-place ‘import’“. The really cool thing about this, they document exactly what we’re trying to do, version-control the /etc structure. So instead of importing /etc, we fake it.

Whilst it’s not documented in the FAQ there, the –parents option is similar to that of the normal mkdir, it recursively creates the directory structure down to the child we’re making. In this case, trunk/etc. After the last 2 commands, you should see a stream of data zip by you. In the above example, I added the entire content of /etc, you can restrict it to certain directories if you want, say apache2, or samba, or you can go the whole way, and do the above.

Now it’s all checked in, lets see what it looks like:

After that, you should see a nice structured view of your checked in directory structure.

A gotcha with SVN

Before we go messing with these files, and testing everything is working well, we need to note something about SVN, that is important to the /etc directory structure. File permissions. SVN does not record file permissions. This is an issue for some parts of the /etc directory, such as the shadow file, or the group file, or that htpasswd file you created to restrict users access.

SVN allows you to store properties information along side each file, this file can contain all kinds of information, such as copyright data, license information, and all that fun stuff. A crafty contributor too advantage of that, and created a script called asvn. This script is a wrapper around the usual svn command, and executes some work either before a commit, or after a checkout. So we need to grab a copy of this script:

After the last command, you’ll see a bunch of stuff zoom past, and you will be prompted for a commit message, I did “permissions fixup”, and hit CTRL X and yes. More stuff zooms by, and you should see a message appear that tells you the revision number has incremented. Now to see if it worked…

Excellent, now the last test, see what the checkout looks like, and see if it matches…

More scrolling of lots of boring text, but you should notice stuff like “checking {somefile} for symlinks”. This is the asvn script doing it’s work, and rebuilding permissions and symlinks. In my example, I’d checked in everything, which includes the precious “shadow” file (I’d not recommend adding that one, add it to the svn:ignore list). I did so to provide a sample file with differing permissions. The original file is locked down to specific user and group:

Lines starting with the $ are the commands I executed. The rest is the response from the server. As you can see, the permissions match.. Now the results in our test checkout…

Permissions match, size matches, so it looks like SVN now has a good copy of the file properties, and the version I have checked in is the same as the one I have in /etc. As a side note, it should be worth remembering that some of the permissions cannot be set by anybody but root, so a lot of this work will probably require root, or sudo access. That being said, you’re editing /etc, so you should already be doing that.

Wrap-up

So far, we’ve covered why we want to use a version control repository for our configurations. We’ve also setup SVN, and got the basics going, including extended SVN to include extra data to handle file properties. Next up is the basics of using SVN, and how it’ll be helping us out here in /etc and configuration land.

If you see any errors, or have any suggestions, feel free to leave them in the comments. Want me to write about something else? Drop me an idea, and I’ll see what I can do.

I’ve been asked to write a post on configuration management, and version control by a friend, Steven Klassen. Instead of a single post, I’m going to break this into several posts as some parts might be unimportant to some, and they can easily skip a whole post. I’ll be posting this over a few days, so stay tuned. If you’re not subscribed to my RSS feed, now would be a great time.

The Request

Earlier today, Steven asked if I used any form of version control. It was a leading question, straight into “dev only or system files”? His questions were targeted, he was searching for guidance. Another question or two later, and it boils down to this:

• Do you use version control for system files?
• Files like domain zone files?
• What about multi-user environments?
• Can you write it up?

My take away on the questions boils down to two things; configuration management, and backups. Anybody that has managed any type of system has, at some point, messed up the configuration file, and forgotten what they did to get there in the first place. Those who have been at it a while will usually do a quick copy before they do any modifications. But, as Steven found out in the past, best laid plans can be foiled by a simple typo.

Version Control

This is what version control is all about. It acts as a backup, whilst keeping a running history of what you did between changes. Version control, or revision control as it’s also known, is big business. Anywhere you find a big name development company, you’ll find they’ve probably developed a revision control system to go with whatever they’re offering. Take a look at this Wikipedia entry comparing just a handful of them.

Configuration Management and Version Control

You don’t have to be a genius to figure out how useful version control can be for configuration management, but in case your brain often goes off to lala land like mine, here is a run down on a few reasons you should really consider using it…

• What did I just change again? And why is that service not starting any more? Oh hell…
• 6 months down the road, can you figure out why you changed that line to Apache to start 10 daemons instead of 5? No? Most version control systems allow you to add comments to your commits.
• As above, but why did Fred in your team change that entry back down to 7 a week later?
• In the words of Homer… Doh!! I wasn’t supposed to delete/overwrite that file!

So, there are 4 obvious cases as to why you’d want to use version control.

The Breakdown

As I said at the beginning, I’ll be breaking this down into several parts. I expect them to be something along these lines:

• Introduction
• Setting up SVN and the base repository
• Using your repository
• Educating your team members
• Maintenance and monitoring
• Conclusion, and follow-up ideas.

If you’re interested, keep an eye open for the rest of the series. If not, send me something you’d like to see me write about. I’m open to suggestions, and will probably butcher anything you can throw at me.

The Nubby Admin has a great post on uptimes, and the old fascination of having a large uptime. Okay, you can get your minds out the gutter now, not that kind of up time. We’re talking servers here.

The post covers a hidden fear, and the goods and bads of large server uptimes. A good read, and one you should look at if you’re watching your server rolling over into the third year of being running. I get a mention (well more of a quote), and seem to fall in with the general crowd, large server uptimes are generally bad.

Go read, enjoy, learn something new from the great minds Wesley is surrounding himself with (myself excluded).

Since IIS6, or there abouts, IIS will not host a file that it does not understand, or doesn’t have a MIME handler for. This is a security feature, but can cause some unknown issues…

We use some customer facing training software by Skillsoft. They produce a Java application which allows you to record voice, and screen shows for training, and follow up with a questionnaire after. Unfortunately because it’s Java, when something goes wrong, it’s very blackbox((A term coined to explain that you can see the outsides, but have no idea what’s going on inside)) like, making it hard to diagnose any issues. In this case, the splash screen would load, and that’s where it’d stop.

Fortunately, the Java console does provide enough information to give us a hint as to what might be wrong. When opening a Java application, the little console app appears in the notification area next to the clock. This allows you to see if the app throws any errors. In our case, it was complaining that a file with the extension .properties was missing from the server (IIS was throwing a 404 message). A quick showed it existed in the directory structure.

This is because IIS doesn’t know what a .properties file is, or how it should be treated. This is easily remedied. In IIS manager, either under the site, or at the root level, go to the MIME option, and add it. In this case, the file extension was .properties, and the MIME Type was specified as application/octet-stream. If it’s done at the root server level, it applies to all sites. If you do it on a site, it only applies to the site you changed it on. No resets, or such, required. Now the file is served properly.

Microsoft has this documented on KB326965. Whilst this applies to IIS6, the same applies to IIS7 as well. To save some waiting in IIS7, you can use the appcmd to add the type for you…

This applies the setting at the root server level, and is usually inherited down to all sites.

Bob Plankers of The Lone Sysadmin fame, has an excellent article titled Midnight is Always Tomorrow, which talks about communicating maintenance windows effectively with peers, as well as customers. He describes the confusion between the term of Midnight, and how some people consider it one day, whilst others see it as a different day.

Midnight is 00:00, meaning the start of a new day. Always.

If you’re in doubt, use 00:01. Assume everybody is clueless about time, because they are. For example, a lot of people think in terms of when they go to sleep, not what actual time it is, so if they’re still up at 0200 on Sunday they consider it to be Saturday.

He timed it well, I was thinking of posting about this very subject myself after a similar incident last week. After our QA team had cleared a break-fix release, it was scheduled for that fateful time, on Midnight on Friday. This was discussed at length by several people on Monday. After several meetings, I hear somebody walking past, telling the business units on Wednesday…

Yes, it’ll be released tonight, and ready for tomorrow.

“Tonight” would have been Wednesday night, tomorrow would have been Thursday. Somehow the magical shift of midnight pushed the release forward a day. Fortunately we interjected, and had them corrected.

This isn’t the only incident that is like this, we see it frequently when something gets requested for that time. I cannot begin to count the number of times I’ve had somebody ask me “is the release tonight at 2am?”. This goes into the idea that people consider “tonight” to be anytime before they went to sleep.

Going with Bob’s suggestion, use dates, and 24 hour clocks, if necessary (and always recommended) shift the time by a few minutes to make it obvious which date midnight falls on. Notify customers, tell them the outage/maintenance/changes are at 00:05 on 06/15/2010 (if you work with international customers, use words instead of numbers).

Bob finishes on a perfect example…

“The system shutdowns will commence at 2200 on 4/17/2010, the power will be disconnected at 0000 on 4/18/2010, and power-ups will occur again at 0800 on 4/18/2010. All times are in CDT (-0500).”

Be clear. Be concise. Be exact. Remove as much ambiguity as you can. The customers will be happy for it.

Whilst sitting here on a product release call, I was reading over my twitter feed, specifically looking at #powershell, and I saw a post by @tonjoh. Feeling I had a few minutes on my hands whilst QA bashed away at the servers, I decided to take a look…

I’m not sure what WMI counter is being used as I cannot find one called free-diskspace-query, but I do know there is a Win32_LogicalDisk, which has enough information in it to tell us free space. So here we go… Lets just get the information we want first, and build up from there…

This is pretty simple, and can probably be found as an example all over the place. For the curious, the output looks like this:

So we can see the size (~111GB), and the amount of free space (~45GB). Now onto the fun stuff, using expressions.

Expressions allow us to do calculations on the resultset that came back from the pre-piped data. This can be something like calculating a percentage, or throwing in miscellaneous data (like requested, a timestamp).

In the above code, I’ve used the back-tick (`) to wrap the lines, and make it easier to read. PowerShell understands this, and will treat the 4 lines as one. I’ve created 2 extra, named, columns, one PercFree, and the other ReportDate. The expression in the first is a calculation based on the values from the Win32_LogicalDisk output, whilst the expression for the second is just extra data in the form of a formatted date. With the above code, you get the following…

And there we have it, two extra columns, which were calculated on the fly.