In my previous post about PowerShell and BITS, I stumbled on a weird quirk in the ForEach-Object function, which had me scratching my head for a bit.

In most programming languages, where you have a for, loop, while, foreach, or other such loop, there is often a set of control functions that go with it. Continue and Break. These functions alter the way a loop is behaving at that time. A call to Continue will stop the current loop iteration at that point in the code, and jump onto the next. A simple example would look like this:

This code simply takes the variable $i, sets it to 0, and whilst $i is less than 5, it loops through the code, each iteration increasing the value of $i by 1. The if statement executes special code when $i gets to the number 2. Now if we run this code, we get the following:

What happened was the counter hit 2, and the code said that it was done with this iteration, and continue processing the outer loop. A Break on the other hand just tells the loop it’s done processing, and to stop looping, and get out. Using the same example as above, but changing Continue to Break we get the output that looks like this:

This is because PowerShell has been told to stop processing any further.

Now for the quirk. I discovered that ForEach-Object isn’t actually a programmatic keyword, it’s a cmdlet. How does this affect the usage of the control functions? Lets look at my code from my BITS article, and compare it to what I had.

As you can see, I’m using the continue function inside the array of strings for the images. If the string value is empty, I skip onto the next one. This is what the original code looked like:

Whilst the code seems pretty similar, there is one distinct difference. The first block of code is inside a loop, the second set is not. Even though you think that ForEach-Object seems like a loop, it’s a cmdlet that does not behave the same way that a loop does. How does this affect the behavior of the code? Badly! Because there is no loop, the control function looks for the next operation it can operate on. It turns out that the next function it can operate on is the entire script. It essentially kills the script right where the line is. This means no processing of other objects in the ‘loop’, no continuing with the code further in the script, no nice handling of anything else in the script. It’s done, finished, over. My hint something wasn’t behaving right was the fact I should have had several thousand images, and yet I only had about 150. It took 4 or 5 attempts, as well as various debug statements throughout the code to figure out why this was happening. After I realized where it was going wrong, a Google search dropped me over to James Manning’s post PowerShell gotcha – foreach keyword vs. foreach-object cmdlet.

So if you ever find yourself in need of an object loop, remember that ForEach-Object isn’t really a keyword, it’s a cmdlet, and does not behave the same.

See any mistakes? Want to add your feedback? Leave me a note in the comments, I love to hear from you.

Ever had to download a bunch of files from a website, and didn’t want to have to write an HTTP handler for it? This comes up on a regular basis here. We get requests from customers that have transferred from one vendor to another, and want to import all their photos to us. Here is how I solve it.

What is BITS?

BITS, or Background Intelligent Transfer Service, is a service bundled with Windows that is used to transfer files from websites. It was originally introduced in Windows XP RTM back in October 2001. You may not know it, but your computer uses it all the time if you have Windows Automatic Updates enabled. It is an intelligent service that asynchronously transfers files, automatically adjusting bandwidth usage, throttling downloads, as bandwidth becomes available. It can even be used to download files across computer starts.

How is it useful in PowerShell?

BITS is really easy to use, a few commands which I’ll show shortly, and you can download a bunch of files quickly, without having to write HTTP handler. In my case above, I receive text files, with a list of image URLs, that need to be imported.

Here is how to do a very simple, single file, transfer.

Relatively simple, a few lines of code for a single file. However, BITS really shines when you’re working with multiple files to download.

As the files I receive are quite often comma-separated values (CSV) files, we can use the rather handy Import-CSV function, and we build up from there.

This is a little more complicated, but is still pretty simple. It introduces Add-BitsFile which allows you to add files to an existing job, and a simple ForEach loop to go through an array of URLs.

I’ve made a few assumptions here, and thrown out all kinds of error handling in favor of a quick code turn around. For example, the job state of a BITS job can be one of nine options (which you can see here), but I assume only two. I’m also not handling cases where I exceed the number of jobs, but that never happens because I only ever have one job running at a time. I’ll probably work on tidying it up a little for other people’s use, but for now, it’s just me using it where I work, so the impact of an error is very minimal.

Give it a shot, download some images off of Flickr, or go a little further, and test it against downloads from Microsoft.

Got any questions? Hints? Tips? Leave them in the comments, I’d love to hear from you.

Whilst driving back from our 4th of July celebrations in San Antonio, I got alerted to a server running low on disk space.  As it hadn’t hit critical, and there was still a fair bit of space left, I decided to wait until I got home.  When I looked at the server, I pulled up my trusted space analyzer, TreeSize.

A quick analysis of the drive, and a report showed that there was a hefty chunk of space lost to the WinSXS directory, and 8GB of Files in the root of C:.  This was a little odd, I know I usually use c:\temp from time to time, but never the root.  To find out what was there, I first had to show hidden and system files.  To do this, go to Tools, then Folder Options, then onto the View tab.  There are 3 options there that we need to change.

1. Change “Hidden files and folders” to “Show hidden files and folders”
2. Uncheck “Hide extensions for known file types”
3. Uncheck “Hide protected operating system files (Recommended)”

I highly recommend doing 2 all the time, you never know when somebody might sent a .txt.exe file.  Changing 3 will prompt that it isn’t recommended, but do it anyway, the files I was having issues with were still hidden after changing the first two.

After making these changes, two files appeared that were the culprits to the lost disk space.  Pagefile.sys and Hiberfil.sys.  I’ll leave the explanation of the Pagefile to the experts.  Hiberfil.sys is a file that is used to handle the state of memory, and running applications when you hibernate your computer. 

What is hibernation?

Hibernation is a heavy sleep state for computers where all the running processes are frozen, the memory is locked, and all the states are saved into that file, then the computer is powered down.  When the computer is turned back on, the operating system sees this state file, and starts to reload everything from there.  This makes it appear like the computer was never even turned off.  The applications are resumed, everything is back to the state it was before hibernation was kicked off.

Why is that bad?

Generally speaking, it isn’t.  In fact, it’s a great thing for home computers.  When you’re done reading your email at night, if you hibernate your computer, and hop into bed, your computer will be off, saving power.  However this isn’t so good for servers.  In 95% of cases, servers don’t ever need to hibernate.  They usually have tasks, and operations that they need to run all the time, which means putting them in a state of hibernation is a bad thing as the tasks won’t be run.

How do you stop this?

Usually, servers have this feature turned off, but for some reason this particular one had it enabled.  I’ve done some looking around, and it seems it’s may have actually been a default option to have it enabled, but for some (lucky) reason a handful of our other servers don’t have it on.  In earlier versions of Windows, disabling the hibernation option was a case of removing a check mark from a box.  This was done in the Control Panel under Power Options, and the Hibernate tab.  However, in Windows 2008 this option was removed, and you now have to use a command line to manage this advanced option.

Within about 10 seconds of issuing that command, the Hiberfil.sys file disappeared, and I recovered 4GB of disk space.  You can read more about the powercfg command here.

I’ve been asked to write a post on configuration management, and version control by a friend, Steven Klassen. Instead of a single post, I’m going to break this into several posts as some parts might be unimportant to some, and they can easily skip a whole post. I’ll be posting this over a few days, so stay tuned. If you’re not subscribed to my RSS feed, now would be a great time.

The Request

Earlier today, Steven asked if I used any form of version control. It was a leading question, straight into “dev only or system files”? His questions were targeted, he was searching for guidance. Another question or two later, and it boils down to this:

• Do you use version control for system files?
• Files like domain zone files?
• What about multi-user environments?
• Can you write it up?

My take away on the questions boils down to two things; configuration management, and backups. Anybody that has managed any type of system has, at some point, messed up the configuration file, and forgotten what they did to get there in the first place. Those who have been at it a while will usually do a quick copy before they do any modifications. But, as Steven found out in the past, best laid plans can be foiled by a simple typo.

Version Control

This is what version control is all about. It acts as a backup, whilst keeping a running history of what you did between changes. Version control, or revision control as it’s also known, is big business. Anywhere you find a big name development company, you’ll find they’ve probably developed a revision control system to go with whatever they’re offering. Take a look at this Wikipedia entry comparing just a handful of them.

Configuration Management and Version Control

You don’t have to be a genius to figure out how useful version control can be for configuration management, but in case your brain often goes off to lala land like mine, here is a run down on a few reasons you should really consider using it…

• What did I just change again? And why is that service not starting any more? Oh hell…
• 6 months down the road, can you figure out why you changed that line to Apache to start 10 daemons instead of 5? No? Most version control systems allow you to add comments to your commits.
• As above, but why did Fred in your team change that entry back down to 7 a week later?
• In the words of Homer… Doh!! I wasn’t supposed to delete/overwrite that file!

So, there are 4 obvious cases as to why you’d want to use version control.

The Breakdown

As I said at the beginning, I’ll be breaking this down into several parts. I expect them to be something along these lines:

• Introduction
• Setting up SVN and the base repository
• Using your repository
• Educating your team members
• Maintenance and monitoring
• Conclusion, and follow-up ideas.

If you’re interested, keep an eye open for the rest of the series. If not, send me something you’d like to see me write about. I’m open to suggestions, and will probably butcher anything you can throw at me.

The Nubby Admin has a great post on uptimes, and the old fascination of having a large uptime. Okay, you can get your minds out the gutter now, not that kind of up time. We’re talking servers here.

The post covers a hidden fear, and the goods and bads of large server uptimes. A good read, and one you should look at if you’re watching your server rolling over into the third year of being running. I get a mention (well more of a quote), and seem to fall in with the general crowd, large server uptimes are generally bad.

Go read, enjoy, learn something new from the great minds Wesley is surrounding himself with (myself excluded).

Since IIS6, or there abouts, IIS will not host a file that it does not understand, or doesn’t have a MIME handler for. This is a security feature, but can cause some unknown issues…

We use some customer facing training software by Skillsoft. They produce a Java application which allows you to record voice, and screen shows for training, and follow up with a questionnaire after. Unfortunately because it’s Java, when something goes wrong, it’s very blackbox((A term coined to explain that you can see the outsides, but have no idea what’s going on inside)) like, making it hard to diagnose any issues. In this case, the splash screen would load, and that’s where it’d stop.

Fortunately, the Java console does provide enough information to give us a hint as to what might be wrong. When opening a Java application, the little console app appears in the notification area next to the clock. This allows you to see if the app throws any errors. In our case, it was complaining that a file with the extension .properties was missing from the server (IIS was throwing a 404 message). A quick showed it existed in the directory structure.

This is because IIS doesn’t know what a .properties file is, or how it should be treated. This is easily remedied. In IIS manager, either under the site, or at the root level, go to the MIME option, and add it. In this case, the file extension was .properties, and the MIME Type was specified as application/octet-stream. If it’s done at the root server level, it applies to all sites. If you do it on a site, it only applies to the site you changed it on. No resets, or such, required. Now the file is served properly.

Microsoft has this documented on KB326965. Whilst this applies to IIS6, the same applies to IIS7 as well. To save some waiting in IIS7, you can use the appcmd to add the type for you…

This applies the setting at the root server level, and is usually inherited down to all sites.

Whilst sitting here on a product release call, I was reading over my twitter feed, specifically looking at #powershell, and I saw a post by @tonjoh. Feeling I had a few minutes on my hands whilst QA bashed away at the servers, I decided to take a look…

I’m not sure what WMI counter is being used as I cannot find one called free-diskspace-query, but I do know there is a Win32_LogicalDisk, which has enough information in it to tell us free space. So here we go… Lets just get the information we want first, and build up from there…

This is pretty simple, and can probably be found as an example all over the place. For the curious, the output looks like this:

So we can see the size (~111GB), and the amount of free space (~45GB). Now onto the fun stuff, using expressions.

Expressions allow us to do calculations on the resultset that came back from the pre-piped data. This can be something like calculating a percentage, or throwing in miscellaneous data (like requested, a timestamp).

In the above code, I’ve used the back-tick (`) to wrap the lines, and make it easier to read. PowerShell understands this, and will treat the 4 lines as one. I’ve created 2 extra, named, columns, one PercFree, and the other ReportDate. The expression in the first is a calculation based on the values from the Win32_LogicalDisk output, whilst the expression for the second is just extra data in the form of a formatted date. With the above code, you get the following…

And there we have it, two extra columns, which were calculated on the fly.

Whilst working away this morning, I received a very pitiful look. It’s that look that we all know. The “I did something terribly wrong” kind of look. This look came with the hands of a digital camera, a small compact point and shoot. A story followed the look, as the camera was handed over to me…

You’ve done it before, I know you have… I don’t know what happened, if it was me, or my daughter, but all my pictures are gone. We went to this event, and I had a lot of pictures, and now they’re all gone. Can you save them…?

Fortunately for me, I know the operation for files deletes.

Anatomy of a File Delete Operation

What a lot of people seem to believe is when a file is deleted, the actual data is erased from the disk. This isn’t actually true in most cases. Really what happens behind the scene is the pointer that tells your operating system where the file is, is removed, or marked as “available”. Essentially the index, that tells your computer where the file is, is removed. Imagine taking whiteout to the index page in a reference book and removing information from there. Now if you were to read every page in the reference book, you’d still find the data, but there is no “quick find” to get there.

How The Day was saved

First… Have a backup, you should have backups, but in this case, the pictures hadn’t even been taken off the camera, so chances of the person making a backup was slim, even though I knew it was likely to be slim even after it was off the camera.

That’s my backup rant out the way (again), now onto the fix-it. Fortunately the camera used a storage card, so I popped it out, and put it into my card reader. I then downloaded a copy of Recuva. This is freeware, but it’s not the only one, there are quite literally hundreds of different applications. Not only is it not the only one, but most of the big name flash card manufacturers have their own software. Once Recuva was opened, you point it to the flash card, tell it to scan, and in about 2 seconds, it’d finished reading the data on the card. Unlike normal file access, this actually reads the bites on the card, rather than relying on the storage card to tell us about. Once finished, you select the files you want recovered, and click the recovery button… Tada, all saved! 300+ pictures in fact, several dating back to Christmas.

How it works…

The JPG format is fairly easy to scan for. There are certain markers in the file that define the start of the image, various bits of data about it (known as the EXIF data, which includes things like date taken, camera used etc), the actual image data, and an end of image marker. When the file recovery applications scan, they look for these markers, and based on the markers they find, they should be able to identify if they can recover the image or not. When it does the recovery steps, it reads all the bytes, from start to end, and writes to a new location. This lets the operating system handle all the usual fun stuff like those indexes/references, and you have your file back.

Caveats

Now there are caveats to this whole process.

First, it assumes you didn’t use a special delete program to remove the file, like Eraser, which does more than deletes the links, it goes back and writes random “stuff” over where the data actually was.

Second, it assumes the system writing to the disk/memory/flash prefers completely blank space, over the space that just had data. Most platforms prefer this because it causes less file fragmentation.

Lastly, this should always be considered your last resort. It’s easy for parts of the file to become overwritten because of other writes on the disk/memory/flash. So as soon as you’ve realized you deleted it, get to work recovering it, don’t wait until next week.

And really, finally… Backup, backup, backup. Did I remember to tell you to backup? No? Okay, go backup. Think it’s too hard? Go read my post on Home Backups, and spend $50/year and get somebody else to do it. I’m sure those memories of little Timmy’s first steps are worth it, or that winning hit at the Texas Rangers game that gets published in the news is worth it.

Part of our application at work sends email notifications to various uses when certain criteria are met. This is great, except testing is difficult as you really don’t want to send out mails to valid customers, so we disable relaying outbound, and the local mail server just chomps on it a bit, then decides to not deliver it and drops it to the local file system. This is great, except in a the matter of days, we have 100k message files. This becomes entirely unmanageable with Windows, so PowerShell is here to help…

There are plenty of articles on file management with PowerShell, pretty much all of them revolving around the Get-ChildItem function. This usually works pretty well, but is terrible on memory when it comes to handling this many files, and working with filters (only selecting files that are X days old).

This is when my brain clicked to a PowerShell.com tip a few weeks back about Windows Management Instrumentation (WMI) based filters. I won’t go into the dark recesses of my brain, and try an explain why I have a weird ability to remember some weird function that I used 5 years ago, but I’ll save you some work and introduce you to the WMI class CIM_DataFile. The cool thing about using WMI to do this, you can use the server side filters to do most of the dirty work for you. So lets see the example:

Okay, that was pretty easy… right? That just gets all the files in the folder C:\inetpub\mailroot\badmail on the server yourremoteserver. Now we want to limit it to just keeping the mail from the last 3 days. Before I did into the code that does this, the date format on CIM_DataFile through PowerShell is a little ‘odd’. Don’t believe me? Try this:

You will probably end up with something that looks like this:

Okay, not too odd, it’s really just the date in the format YYYYmmddhhmmss.nnnnnn-tzo. n is milliseconds, and tzo is the timezone offset in minutes. In the above example, 300 would be 5 hours, 360 is 6 hours. This means, for the query, we have to modify slightly.

You might be wondering what that weird stuff on line 3 is, it’s the PowerShell equivalent to C#’s string.Format. This takes the date we had, and reformats it nicely for WMI to use. This is much nicer, gets WMI to do all the grunt work, and is much more friendly on the server (memory/CPU). Next, the delete…

As I mentioned at the beginning, it’s possible to use the Get-ChildItem function to do the same thing (except it’s not remote), so I figured I’d show how too…

The problem you can see here, the filtering is done after all the items are fetched, rather than restricting the fetched list to begin with. This cannot be executed remotely either (unless using network shares), so running this from a central job server is a little more tricky.

I’m sticking with the WMI method for now, it’s fast, memory and CPU friendly, and does the job just as nicely. How do you guys manage remote files with PowerShell? Any hints and tips?

Edit: Thanks to Steven Klassen, I edited some of the code. Apparently the CodeColorer plugin was escaping my escape and only showing a single backslash on the end of some of the commands.

From time to time, I have to share screen shots, it’s a common task for some, especially when showing errors and the likes. This is usually fine because it can be emailed, or saved as an image file for attaching to a bug ticket. I’ve started sharing some screen shots online, which now requires a few more steps, and becomes quite time consuming, screen shot, save image, upload to some site. This is where jing came in…

Jing is an application that can be used to take screen shots, or videos. It has an added bonus that it is backed by Screencast.com. With the two combined, you could take a screenshot, have it uploaded, and get a URL to send to other people in as little as 3 clicks. Pretty cool, especially as you can do video as well.

However, I was struggling with a memory issue, upwards of 300MB after taking a single screen shot. The Jing folks followed up on my tweet with an interesting posting on memory management. It details some interesting stuffs about .NET and the Garbage Collector. Whilst this is a great post, and provides some excellent insight into some application memory management, it doesn’t resolve my memory issues. So I am stubborn, and went looking for some other options.

I stumbling across a site called AlternativeTo, which has references to possible alternatives to applications. A couple of the applications looked interesting, but not quite what I wanted. Then I stumbled upon zscreen. Another .NET application which seems quite a bit less memory hungry. The UI is basic, but stuffed with options. Including not limiting your uploads to just a specific site, with built in support for RapidShare, SendSpace, Flickr, ImageShack, TinyPic, Twitter, ImageBarn, and MindTouch to name a few. That’s not the only option either, with FTP support an option too, you can push to your own private site, and generate URLs for your use.

Whilst jing faithfully served me as the little glowing yellow ball at the top of my screen for a while, zscreen has become my screenshot taker of choice. Give it a shot, worth playing with.