Whilst driving back from our 4th of July celebrations in San Antonio, I got alerted to a server running low on disk space.  As it hadn’t hit critical, and there was still a fair bit of space left, I decided to wait until I got home.  When I looked at the server, I pulled up my trusted space analyzer, TreeSize.

A quick analysis of the drive, and a report showed that there was a hefty chunk of space lost to the WinSXS directory, and 8GB of Files in the root of C:.  This was a little odd, I know I usually use c:\temp from time to time, but never the root.  To find out what was there, I first had to show hidden and system files.  To do this, go to Tools, then Folder Options, then onto the View tab.  There are 3 options there that we need to change.

1. Change “Hidden files and folders” to “Show hidden files and folders”
2. Uncheck “Hide extensions for known file types”
3. Uncheck “Hide protected operating system files (Recommended)”

I highly recommend doing 2 all the time, you never know when somebody might sent a .txt.exe file.  Changing 3 will prompt that it isn’t recommended, but do it anyway, the files I was having issues with were still hidden after changing the first two.

After making these changes, two files appeared that were the culprits to the lost disk space.  Pagefile.sys and Hiberfil.sys.  I’ll leave the explanation of the Pagefile to the experts.  Hiberfil.sys is a file that is used to handle the state of memory, and running applications when you hibernate your computer. 

What is hibernation?

Hibernation is a heavy sleep state for computers where all the running processes are frozen, the memory is locked, and all the states are saved into that file, then the computer is powered down.  When the computer is turned back on, the operating system sees this state file, and starts to reload everything from there.  This makes it appear like the computer was never even turned off.  The applications are resumed, everything is back to the state it was before hibernation was kicked off.

Why is that bad?

Generally speaking, it isn’t.  In fact, it’s a great thing for home computers.  When you’re done reading your email at night, if you hibernate your computer, and hop into bed, your computer will be off, saving power.  However this isn’t so good for servers.  In 95% of cases, servers don’t ever need to hibernate.  They usually have tasks, and operations that they need to run all the time, which means putting them in a state of hibernation is a bad thing as the tasks won’t be run.

How do you stop this?

Usually, servers have this feature turned off, but for some reason this particular one had it enabled.  I’ve done some looking around, and it seems it’s may have actually been a default option to have it enabled, but for some (lucky) reason a handful of our other servers don’t have it on.  In earlier versions of Windows, disabling the hibernation option was a case of removing a check mark from a box.  This was done in the Control Panel under Power Options, and the Hibernate tab.  However, in Windows 2008 this option was removed, and you now have to use a command line to manage this advanced option.

Within about 10 seconds of issuing that command, the Hiberfil.sys file disappeared, and I recovered 4GB of disk space.  You can read more about the powercfg command here.

Since IIS6, or there abouts, IIS will not host a file that it does not understand, or doesn’t have a MIME handler for. This is a security feature, but can cause some unknown issues…

We use some customer facing training software by Skillsoft. They produce a Java application which allows you to record voice, and screen shows for training, and follow up with a questionnaire after. Unfortunately because it’s Java, when something goes wrong, it’s very blackbox((A term coined to explain that you can see the outsides, but have no idea what’s going on inside)) like, making it hard to diagnose any issues. In this case, the splash screen would load, and that’s where it’d stop.

Fortunately, the Java console does provide enough information to give us a hint as to what might be wrong. When opening a Java application, the little console app appears in the notification area next to the clock. This allows you to see if the app throws any errors. In our case, it was complaining that a file with the extension .properties was missing from the server (IIS was throwing a 404 message). A quick showed it existed in the directory structure.

This is because IIS doesn’t know what a .properties file is, or how it should be treated. This is easily remedied. In IIS manager, either under the site, or at the root level, go to the MIME option, and add it. In this case, the file extension was .properties, and the MIME Type was specified as application/octet-stream. If it’s done at the root server level, it applies to all sites. If you do it on a site, it only applies to the site you changed it on. No resets, or such, required. Now the file is served properly.

Microsoft has this documented on KB326965. Whilst this applies to IIS6, the same applies to IIS7 as well. To save some waiting in IIS7, you can use the appcmd to add the type for you…

This applies the setting at the root server level, and is usually inherited down to all sites.

Whilst working away this morning, I received a very pitiful look. It’s that look that we all know. The “I did something terribly wrong” kind of look. This look came with the hands of a digital camera, a small compact point and shoot. A story followed the look, as the camera was handed over to me…

You’ve done it before, I know you have… I don’t know what happened, if it was me, or my daughter, but all my pictures are gone. We went to this event, and I had a lot of pictures, and now they’re all gone. Can you save them…?

Fortunately for me, I know the operation for files deletes.

Anatomy of a File Delete Operation

What a lot of people seem to believe is when a file is deleted, the actual data is erased from the disk. This isn’t actually true in most cases. Really what happens behind the scene is the pointer that tells your operating system where the file is, is removed, or marked as “available”. Essentially the index, that tells your computer where the file is, is removed. Imagine taking whiteout to the index page in a reference book and removing information from there. Now if you were to read every page in the reference book, you’d still find the data, but there is no “quick find” to get there.

How The Day was saved

First… Have a backup, you should have backups, but in this case, the pictures hadn’t even been taken off the camera, so chances of the person making a backup was slim, even though I knew it was likely to be slim even after it was off the camera.

That’s my backup rant out the way (again), now onto the fix-it. Fortunately the camera used a storage card, so I popped it out, and put it into my card reader. I then downloaded a copy of Recuva. This is freeware, but it’s not the only one, there are quite literally hundreds of different applications. Not only is it not the only one, but most of the big name flash card manufacturers have their own software. Once Recuva was opened, you point it to the flash card, tell it to scan, and in about 2 seconds, it’d finished reading the data on the card. Unlike normal file access, this actually reads the bites on the card, rather than relying on the storage card to tell us about. Once finished, you select the files you want recovered, and click the recovery button… Tada, all saved! 300+ pictures in fact, several dating back to Christmas.

How it works…

The JPG format is fairly easy to scan for. There are certain markers in the file that define the start of the image, various bits of data about it (known as the EXIF data, which includes things like date taken, camera used etc), the actual image data, and an end of image marker. When the file recovery applications scan, they look for these markers, and based on the markers they find, they should be able to identify if they can recover the image or not. When it does the recovery steps, it reads all the bytes, from start to end, and writes to a new location. This lets the operating system handle all the usual fun stuff like those indexes/references, and you have your file back.

Caveats

Now there are caveats to this whole process.

First, it assumes you didn’t use a special delete program to remove the file, like Eraser, which does more than deletes the links, it goes back and writes random “stuff” over where the data actually was.

Second, it assumes the system writing to the disk/memory/flash prefers completely blank space, over the space that just had data. Most platforms prefer this because it causes less file fragmentation.

Lastly, this should always be considered your last resort. It’s easy for parts of the file to become overwritten because of other writes on the disk/memory/flash. So as soon as you’ve realized you deleted it, get to work recovering it, don’t wait until next week.

And really, finally… Backup, backup, backup. Did I remember to tell you to backup? No? Okay, go backup. Think it’s too hard? Go read my post on Home Backups, and spend $50/year and get somebody else to do it. I’m sure those memories of little Timmy’s first steps are worth it, or that winning hit at the Texas Rangers game that gets published in the news is worth it.

Every now and again, we have issues where a developer will have their laptop rebuilt, and forget they have files checked out in Team Foundation Server (TFS). This isn’t very helpful as sometimes they cannot undo their check outs. Visual Studio doesn’t give you access to undo a checkout in another workspace if it’s not your workspace, however the TF.exe command does. Here’s how:

This will happily undo the change on that particular file. If you have a whole directory to revert the changes on, that’s easy too…

You must remember when reverting somebody else’s change to use the project URL, not the physical path on your local machine, otherwise you’ll get an error like this:

For more details on the tf.exe undo command, see the Microsoft documentation here. Full details on arguments for the tf.exe command can be found here too.

This one is a quick and easy one. We have multiple IIS boxes, which each generate large usage logs on a daily basis. To save space, and for analytics off-server, we compress the log files. Having found Rar is one of the better compression algorithms, we simply shell out to rar, compress the logs, and cleanup older logs.

This could probably do with some cleaning up as there are probably some slightly more efficient ways of doing some of this, but it works.

We’ve had a stubborn Windows 2003 server that we’ve been trying to get Office 2007 installed for a while. It has a couple of things wrong with it which have made the install substantially harder.

The first being an MSI speed issue. Because the server plays host to about 120+ custom applications, all installed using MSI, any further MSI installs simply drag whilst it reads through all the keys, does its validation and backup checks, and then proceeds.

The second issue we have is that this box has a partially installed, corrupted Office 2007 install already on here. Attempting to do “upgrades” fail, and uninstalling just barfs with a delightful error. This resulted in some stumbling to figure out how to get it off, and start again.

Fortunately Microsoft do frequently document things like this, you just have to find the right search terms to find it, which luckily I did, in the form of KB928218. This article gives you a list of the registry keys you have to go through and remove, and you should be all well and good. Unfortunately, this list is over 200 registry keys on our count, and quite a list all over the place. This is where Powershell comes in handy…

As the server itself doesn’t (yet) have Powershell installed, I ran the commands from my laptop, which required remote registry access. Powershell makes use of the .Net libraries to do this:

The LocalMachine reference here is the part of the registry we want to open, which is equivalent to HKEY_LOCAL_MACHINE. This now gives us a registry object in the HKLM section of the registry. From here, there are a bunch of functions we can use to fetch the keys we need. The first is OpenSubKey. This opens a subkey in the $reg object.

This assigns $regKey to the subkey. The boolean flag $true on the end is to make sure the key is opened in writable mode. This will be required to modify the contents of the key.

Per the KB article, Microsoft say to delete all of the 12.0 subkey from under here. I had originally used the following:

This resulted in an error about not being allowed to delete subkeys. Then I stumbled upon the much more suitable DeleteSubKeyTree, which, as the name hints, deletes the tree. So the new code became:

This was repeated for the various other subkeys Microsoft recommended deleting. It became more complex when it asks you to delete keys that match some of the characters below, and use a * as the match. For example:

Note In the following registry keys, the asterisk character (*) represents one or more characters in the subkey name. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\*0FF1CE*

Fortunately, Powershell is very good at string manipulation, and queries, so this was a bit of a case for a where, and foreach loop.

It looks pretty messy, but it’s actually pretty easy if you understand pipes (the | character). The above line 24 boils down to:

• Get the subkey names
• Find only the ones that have 0FF1CE in them (yes I know that’s office with a zero and a one)
• For each one that matches, go delete it, and its children

See, not so hard.

There are some parts of this that venture KB into other areas of the registry, so the syntax changes in the opening command, for example:

This opens HKEY_CLASSES_ROOT. You can use CurrentUser to access HKEY_CURRENT_USER too.

That’s about it for remote registry access and fixing this blasted Office install. As a side, the script saved me about 30 minutes of bashing through about 200+ registry keys for cleaning up.

Below is the full script, with the delete statements commented out. If you have 2007 installed, give it a run, see what happens (please double check that all the deletes are in fact commented out, this script was used against a server to fix an install issue, and did have active delete statements in, and I cannot remember if I got all the deletes).

The Networker Blog has an excellent post on support contracts, coining the term Icarus Support Contract. Preston warns on the dangers of using a using minimal support agreements when covering equipment, and software in an environment that is covered by an SLA.

Preston defined an Icarus support contract as…

Well, it’s a contract where you rely on luck. It’s a gamble – that in the event of a serious problem, you can buy immediate assistance at the drop of a hat.

I think this is a great post, and an excellent example of why people probably need to step back, and address their support needs, and agreements. I can even cite a personal example from recently. During out data center migrations, as part of the company transition to a new parent company, the support agreement slipped on several devices. They were expired by, quite literally, a month. During that time, 2 of the devices had physical hardware failures, and required an RMA to resolve. Unfortunately because they were out of contract, it would have cost us a fortune. Fortunately enough, the vendor was good enough to work with us, and get it sorted, they were even good enough to drop the re-up fees required on the other devices. Had we had the support contracts in place, we’d have had a device on site the next day, instead we ended up waiting nearly 2 weeks due to dealing with contracts, approvals, and signatures. Fortunately the devices were not scheduled to be deployed for another 3 months… But what were to have happened if they were live systems? I think our customers would have been rather upset about the 2 week outage.

So do you have enough of a support agreement to cover your equipment, both software and hardware, to cover your SLAs? If not, ask yourself how much it’ll cost you to get there, and if it’s worth the initial expenditure, or the cost of customer loss. If you don’t have an SLA, do your customers have an expectation of you, and your services? If so, now is the time to establish a well documented, and agreed upon, SLA, and make sure your support agreements are up to scratch to help cover the SLAs.

For the last day or so, we’ve been working on rolling out the next release of our code to our QA environment. When it came time to build a new set of solutions I stumbled across a weird validation error.

This error popped up on the setup applications for several new programs. Doing some searching I stumbled across a couple of posts that hinted on removing the project output from the setup, and trying again. This stopped the errors on the setup application, but not the actual application itself.

Then I saw a hint in the errors list…

Checking into the app.config file, I stumbled across the cause. It looks like somebody had pasted code that had been “fixed” up by Outlook, or word, and broken a comment. Fixing this, resolved both the app compilation, and the setup validation error.

Part of our move was including the building of the new DR environment. Like most people, we cheated, and cloned the production servers to the new environment. One issue was host naming changes. In our DR environment, the host names change due to the location, and roll of the servers. This means that various IIS config options are now incorrect. This is easily fixed with the new AppCmd in IIS7.

The command is relatively easy to understand. /vdir.name is the path to the site, and virtual directory, and /physicalPath is the new path. Easy huh?

I’m obviously playing catch up with a number of posts I’ve been meaning to do. This was something from something I read back in November by Tom Limoncelli, but was something I had planned on writing about anyway. The post, titled “Run, run, run, dead“, brings a nice analogy of things breaking in the analog/digital eras, and points out that as system administrators, we should be using the analog method of monitoring.

An analog radio (one with an old-fashion vacuum tube) sounds great at first, but you hear more static when the tube starts to wear out. Then the tube dies and you hear nothing. If you change the tube when it starts to degrade, you’ll never have a dead radio. (Assume, of course, you change the tube when your favorite radio show isn’t on.) A transistor radio, on the other hand, is digital. It plays and plays and plays and then stops. Now, during your favorite song, you have to repair it.

This is one of those great analogies that works well for the situation. Watching for the host to go down is reactive, it’s already too late (transistor radio), watching for stuff changing, and adapting your system(s) to it is proactive. Take for example the image to the right. This is Nagios monitoring our VPN server’s connection counts. The server was in a small subnet, with a restricted number of connections (30). Nagios warns to excessive connections at 25, and alerts critical at 27. This allows us to take a look, and bump duplicate connections, or excessive timed connections. This was a temporary solution, because like any good admin, we didn’t want to sit messing with peoples’ connections all the time, so we moved the server so we could give it a bigger block. In this scenario we reacted to the changing demand of the VPN server as we watched trends (the year graph shows a trend upwards from 10-15 up to the current 25-28).

This is just one example of a reactive monitor, we watch all kinds of metrics from memory, to disk space, to IO, to webserver connections, all the way through to bandwidth utilization on our connections and VPN tunnels.

Reactive is fine for up/down status, but no good way to plan for your future needs, or your current performance trends. If you’ve not read either of Tom’s books, and you’d a system administrator, you should.