Downloading Files with PowerShell

While messing around trying to diagnose an issue with IIS and compression yesterday I had a need to download a whole bunch of files all at once (or at least in quick succession).

Calling in the libraries from the .NET framework; this task is actually really easy.

$src = "http://localhost/test.cmp"
$dst = "F:\Downloads\junk\test_{0}.cmp"
$web = New-Object System.Net.WebClient

$web.Headers.Add([System.Net.HttpRequestHeader]::AcceptEncoding, "gzip")

1..100 | %{
	$web.DownloadFile($src, $dst -f $_ )
}

The script is relatively self-explanatory. $src and $dst are the source and destination files. For the destination I’ve used a formatted string allowing me to inject values into the string using C# style formatting. I create a new object using the System.Net.WebClient type. I then loop 100 times and download the same file, saving it to a different destination name each time.

The above code includes setting an “Accept-Encoding” header to the request. Because I was testing a gzip compression issue on IIS I needed this header, otherwise WebClient just requests the raw data. The caveat to this is that the file being written out to the $dst path is actually a gzip compressed file and not the actual data. This is fine for my testing because it made it quick and easy to see if compression had worked. I was downloading a file containing 300KB of Lorem Ipsum text. If it the IIS compression worked the file would be much smaller. I’ll do another blog post soon about handling the gzip data and turning it back to the same as the source file.

I could have done the same using the BITS service, but BITS would probably have fixed the gzip’d file so I’d have to do more work to determine if compression had actually worked (examining logs, network trace, IIS trace, etc). This worked out nicely.

As a side note, the “1..100” code is a shortcut for generating a sequence of numbers from 1 to 100. If you opened a powershell prompt, typed “1..100” and hit enter it’d spew out all the digits from 1 through to 100. Passing this on to foreach-object we can turn it into a loop. The other way to do this would be to use a for, do, or while loop.

for( $i=0; $i -lt 100; $i++) {
  $web.DownloadFile($src, $dst -f $i)
}

$i=0
do {
  $web.DownloadFile($src,$dst -f $i)
  $i++
} until ($i -eq 100)

$i = 0
while ($i -lt 100) {
  $web.DownloadFile($src, $dst -f $i)
  $i++
}