TheGeekery

The Usual Tech Ramblings

Downloading Files with PowerShell

While messing around trying to diagnose an issue with IIS and compression yesterday I had a need to download a whole bunch of files all at once (or at least in quick succession).

Calling in the libraries from the .NET framework; this task is actually really easy.

$src = "http://localhost/test.cmp"
$dst = "F:\Downloads\junk\test_{0}.cmp"
$web = New-Object System.Net.WebClient

$web.Headers.Add([System.Net.HttpRequestHeader]::AcceptEncoding, "gzip")

1..100 | %{
	$web.DownloadFile($src, $dst -f $_ )
}

The script is relatively self-explanatory. $src and $dst are the source and destination files. For the destination I’ve used a formatted string allowing me to inject values into the string using C# style formatting. I create a new object using the System.Net.WebClient type. I then loop 100 times and download the same file, saving it to a different destination name each time.

The above code includes setting an “Accept-Encoding” header to the request. Because I was testing a gzip compression issue on IIS I needed this header, otherwise WebClient just requests the raw data. The caveat to this is that the file being written out to the $dst path is actually a gzip compressed file and not the actual data. This is fine for my testing because it made it quick and easy to see if compression had worked. I was downloading a file containing 300KB of Lorem Ipsum text. If it the IIS compression worked the file would be much smaller. I’ll do another blog post soon about handling the gzip data and turning it back to the same as the source file.

I could have done the same using the BITS service, but BITS would probably have fixed the gzip’d file so I’d have to do more work to determine if compression had actually worked (examining logs, network trace, IIS trace, etc). This worked out nicely.

As a side note, the “1..100” code is a shortcut for generating a sequence of numbers from 1 to 100. If you opened a powershell prompt, typed “1..100” and hit enter it’d spew out all the digits from 1 through to 100. Passing this on to foreach-object we can turn it into a loop. The other way to do this would be to use a for, do, or while loop.

for( $i=0; $i -lt 100; $i++) {
  $web.DownloadFile($src, $dst -f $i)
}

$i=0
do {
  $web.DownloadFile($src,$dst -f $i)
  $i++
} until ($i -eq 100)

$i = 0
while ($i -lt 100) {
  $web.DownloadFile($src, $dst -f $i)
  $i++
}

DL365 G5 battery location

Sometimes it’s hard to understand hardware design (no pun intended). I can understand there are complex requirements, depending on what’s being designed. Today I stumbled on what I’d call a pretty odd design flaw, but can see why it was done with poor foresight.

In this case, I’m referring to the design of HPs DL365 G5 servers. Particularly the decision for the location of the storage controller’s battery. I understand that 1U servers have some very specific, and complex requirements, such as:

  • Must fit all the stuff in a small space
  • Must have good air flow
  • Must provide easy access to hot-swap parts

Earlier this month we got an alarm on a server that the battery had gone bad, and was no longer holding a charge. I scheduled downtime, made sure we had a battery on hand, put in a change ticket, and set to work.

DL365 Fan After powering down the server, and removing the top cover, I set out to find the battery. The storage controller was next to the memory banks, and had 3 cables coming from it. 2 were obviously for the HDD back planes, the third, a little thinner, disappeared behind some fans, and into the front of the case.

DL365 Fan If you take a look at the picture to the right, and look above the center fan bank, you can see a thin black cable disappearing under the chassis. A closer look is in this picture. From the front, it’s hiding right above the middle drive behind this grill.

To remove the fan bank is a project on its own. Dell uses nice quick release containers, squeeze some colored tabs, and stuff just pops apart. To get these out, I had to use a flathead screwdriver to pry the metal tabs out the way. Well, they’re not really tabs. They were about 2 inches long, and about the same deep, and not flexible in the slightest.

DL365 Storage Controller Batter Once removed, it should be a simple case of pulling on a small black tab. This is where the design flaw comes in (as if the fact its hidden in there to begin with isn’t bad enough). These storage batteries have a tendency to swell as they get older, even more so as they reach the end of their life. If you look at the picture to the left, you can see the bulge already. Why is this a problem? Well 2 reasons, the first being fairly obvious… If it’s bulging, there is a chance for it to explode, or leak, all over the inside of the server, in particular onto the hard drives below it. The second, is the design flaw, the battery is nicely wedged between 2 sheets of rivited metal, and when the battery swells, it does NOT want to come out.

It can take some ‘gentle’ pursuasion to evict the battery from it’s sanctuary without puncturing the first cell on the little board. In this case, I had to use a flat head screwdriver, and go through the grill at the front of the chassis above the drives to push it, while I carefully pulled from the other side at the same time. There is a few more pictures of the swollen battery here, but this one shows it pretty well.

The battery was probably put in that location because it was the only free space left. Little consideration was taken for maintenance, or hardware faults, such as swelling batteries. I believe the G7 series has this issue resolved though.

Still Alive...

It has been a while since I’ve posted. The last was on my string vs bash match up. It got a fair bit of attention, with some great feedback, so I learnt some new stuff. I’m going to try getting back to posting more regular posts, even if they aren’t always techy related1. For now, here’s what has been going on…

  • New job
  • More volunteer work with CERT
  • More volunteer work at school

Generally as busy as before, but at the same time, a different kind of busy.

Job

The transition to a new job was probably one of the harder decisions I’ve had to make in a long time, but now I’ve settled in, I don’t regret it at all. I’m getting my hands on a lot more stuff I didn’t have access to before, as well as taking lead on projects that were originally left to HQ folk.

The environment is a lot larger. I’ve gone from a handful of few racks with gear to 20+ full racks, not counting the volume of VM guests we have running.

Documentation is pretty limited, so I’m working on updating it all, starting with a full data center audit. I’m also working on a Lync pilot, as well as updating monitoring and other projects.

CERT

I’ll have a complete post on this one later. In the meantime, you can read Christopher Webber’s post on CERT and Sys Admins.

With tornadoes striking the DFW metroplex recently, we were asked to help with some cleanup operations. Post and pictures on that later.

So lots of new exciting stuff floating around, learning new stuff, getting my fingers into new toys and such. I’ll post more stuff as it comes up.

  1. I do have some in the works on CERT and Emergency management. 

PowerShell vs Bash... String processing face off

While working on a file cleanup project today, I had to work with a text file containing close to 2 million lines. I had to extract file names with a specific string within it. I figured I could do it relatively quickly in PowerShell, but also realized it’d be a good opportunity to flex my Bash skills. To say the results were interesting is an understatement (to me at least). I picked up some tuning tricks while tinkering, which made some massive improvements.

eclipse, TFS Everywhere, and new files

I work in a mostly windows environment, until relatively recently, where we acquired a new company. The new company was mostly Mac, so I acquired myself a Mac Air. With that, I decided to see how much of my Windows life I could transport over there. I use Microsoft’s Team Foundation Server for managing source control, so I had to find a good alternative to trying to get Visual Studio working on a Mac platform. Fortunately back in 2009, Microsoft acquired a company called Teamprise, which developer a client that worked on a variety of platforms. Microsoft re-released that as TFS Anywhere. I use TFS Anywhere to edit a lot of the files in TFS on my Mac, via Eclipse.

A few weeks ago I stumbled on an odd issue where Eclipse wasn’t keeping in-sync with what TFS Anywhere was doing. For example, we keep our Nagios configs checked into TFS for version control. I was using the Mac, and had added several new contact files. When I was back using my Windows machine, I told Eclipse to trigger the “Get Latest” function on TFS Anywhere, which it dutifully did, then threw an error…

The item $/infrastructure/Nagios/etc/objects/contacts/newcontact.cfg already exists

It repeated this error for each of the new contacts I had added. What appeared to be happening is TFS Anywhere was getting the files, notifying Eclipse the files were there, which triggered the project to add them to the file list, which then in turn triggered Eclipse to try an add them to TFS. A nice little circle going there. I thought maybe it was just a quirk and closed the project. When I reopened the project, the same error occurred again, this time I didn’t even tell Eclipse to do any updates, this occurred just on opening the project.

After some skimming around, I discovered this is likely a bug in TFS Anywhere, and the simple solution is to tell Eclipse to update the server information. This is done using the “Team” context menu (right click the project), then selecting “Refresh Server Information”. Here is the reference post I found the details on.

PowerShell and SQLite

A while back I mentioned I was using SQLite with PowerShell. I was doing this because I had to access the database for gPodder to tweak some of the subscriptions. A need came up again today after upgrading gPodder to the latest release, and having issues with it.

Android Toys

Okay, less toys, and more toy. Not only does our other entity work with Apple devices, we have an Android application. Currently development has been done using personal equipment, which has stunted the development to what few android phones we had available. So we decided to pick up a few more Android devices for them to work with.

IPv6 WAN Party

Joseph Kern has come up with a novel idea, an IPv6 WAN party.

IPv6 is a big project that is going to impact everybody, even if they don’t know it. So this WAN party gives us a chance to get ahead of the curve, and get some much needed knowledge under out belts, while doing practical work. Joseph’s idea is that we all setup IPv6, using Hurricane Electric’s “Tunnel Broker” service, on our home networks, then take the certification courses on http://ipv6.he.net. There is even the possibility of a Google+ Hangout, while we’re working, for moral support.

If you’re interested, follow Joseph on twitter @josephkern or the hashtag #ipv6wanparty. Initial plans look to host it on Saturday November 12th, at 12pm EST.

Windows Update and Error 0x800f0818

Microsoft Windows Update has a track record of having the worst error reporting, usually throwing error codes, rather than helpful errors. I stumbled on another case of this after I worked on 2 identical machines this week. Windows Update reported the unhelpful error code 0x800f0818 during an update of .Net 3.5 SP1.