TheGeekery

The Usual Tech Ramblings

Powershell and Progress Feedback

We’re in the process of enabling a new password reset portal, which requires additional licensing features in Office 365. There is no “apply all” button in Office 365, so we have to do this via script to tens of thousands of user accounts. The problem with this is some form of feedback to the user running the script. PowerShell has some really handy built in features, such as a progress bar. It’s amazing how something so simple can make you feel a little better, and actively see how things are moving along. Progress bars in PowerShell are incredibly simple.

1
2
3
4
5
6
7
$count = 100

for($i = 1; $i -lt $count; $i++) {
       $pctComp = ($i /$count) * 100
       Write-Progress -Activity 'License assignment...' -Status $('{0}% complete' -f $pctComp) -PercentComplete $pctComp
       sleep 5
}

So this is very basic, all I’m doing is setting a counter to 100, and using a variable $i and incrementing the number. Then figuring out the percentage and using it to set the value in the progress bar. The sleep 5 is so we can actually see the progress bar in action. In our real world example, the count is based on the number of user objects, and no sleep was needed because it was actually doing work, unlike our sample code above.

Progress Bar

You can even get fancy with your progress bars, and actually have multiple progress bars. A use case example for this would be if you have a lot of operations on a single object, you might want to report the progress of that object. I’m just going to use a loop inside a loop for my example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$count = 100

for($i = 1; $i -lt $count; $i++) {
       $pctComp = ($i /$count) * 100
       Write-Progress -Activity 'License assignment...' -Status $('{0}% complete' -f $pctComp) -PercentComplete $pctComp -Id 1

       $innerCount = 50
       for ($m = 1; $m -lt $innerCount; $m++) {
             $innerPcgComp = ($m /$innerCount) * 100
             Write-Progress -Activity 'Inner loop' -Status $('{0}% complete' -f $innerPcgComp) -PercentComplete $innerPcgComp -ParentId 1
             sleep 1
      }

       sleep 2
}

In this example, notice how I use a -id and on the inner loop use -parentid. With these, the second progress bar becomes indented as a child of the first progress bar. You can keep going through multiple layers if you feel you want to, or you can have multiple parent loops and multiple child loops.

Progress Bar2

Your status messages don’t have to show the percentage either, they can be messages relating to the location in code. Here is another example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$count = 100

for($i = 1; $i -lt $count; $i++) {
       $pctComp = ($i /$count) * 100
       Write-Progress -Activity 'License assignment...' -Status $('{0}% complete' -f $pctComp) -PercentComplete $pctComp -Id 1

       Write-Progress -Activity 'Inner Loop' -Status 'Starting Inner Loop' -PercentComplete 1 -ParentId 1 -Id 2

       sleep 2

       Write-Progress -Activity 'Inner Loop' -Status 'Doing some other action' -PercentComplete 5 -ParentId 1 -Id 2

       sleep 2

       Write-Progress -Activity 'Inner Loop' -Status 'Doing something else' -PercentComplete 15 -ParentId 1 -Id 2

       sleep 2

       Write-Progress -Activity 'Inner Loop' -Status 'jumping waaaay up there' -PercentComplete 95 -ParentId 1 -Id 2

       sleep 2

       Write-Progress -Activity 'Inner Loop' -Status 'Finishing' -PercentComplete 100 -ParentId 1 -Id 2

       sleep 2
}

Progress Bar3

Progress Bar4

Sometimes it’s the simple things that we can add to a script that gives a whole lot of feedback to the user running it. If I’d not done a simple progress bar, we’d be clueless as to the progress of the script running against 45,000 users, and as the script took hours to run, some form of feedback is critical.

What other methods of feedback do you guys use? Have you used progress bars in a different fashion? Let me know.

Custom Windows Installs, Injecting Drivers and Features

One of the things about new platforms is you get to learn new technology. One of the bad things about new technology is that a lot of your old methods might not apply anymore, need to be revamped, or redesigned completely. Over the last few months I’ve been working on a Cisco UCS platform deployment for work. This has been quite exciting as it’s all new gear, and it’s implementing stuff that we should have been able to implement with our HP BladeSystem C7000 gear.

One of the biggest gotchas so far is that the build image we have for our machines no longer works. The Cisco UCS B200 servers have hardware that isn’t detected out of the box with Windows 2012R2. This means we have to inject drivers into the boot image, and the install image, to make the install work when using Boot From SAN (BFS).

This post is a reflective for myself, and others that might find it handy, because I’m constantly forgetting how to update the drivers in boot images. One thing I’m very thankful for, the new ImageX format that Microsoft started using with Windows Vista. This makes image management so very much easier.

Preparing your build environment

First step is to install Microsoft ADK (Assessment and Deployment Kit). When you run the install, you only need to install the 2 deployment and build packages.

The next step is to prepare the build environment. I have a secondary drive in my desktop, so I built the following structure:

1
2
3
4
5
6
7
8
F:\
|-Build
  |-ISO
  |-Windows
  |-Drivers
    |-Network
    |-Storage
    |-Chipset

You’ll need a valid copy of Windows 2012R21. If you have it on DVD, simply copy the contents of the DVD into your Windows folder as I have documented above. You’ll also need a copy of the driver CD from the vendor. As this is for the Cisco UCS B200, you need a login, and you can find them tucked here.

Find the drivers you need on the DVD, in my case the Network and Storage drivers were easy as there were only one in the named folders. The chipset was a little difficult because a clean install of Windows didn’t detect the chipset, but pointing the Windows at the driver DVD found all the drivers, and then the device said no special drivers were needed so refused to list any. After some fudging around, I managed to identify these as Intel’s Ivytown drivers, so dropped those in the Chipset folder.

Identifying Install Image and Injecting Drivers

This is where all the magic happens. We’re going to inject the drivers into the image, or slipstream them as it’s called in some places. This is done with just a handful of commands. The first thing we need to do is identify which install image we want to work with. A standard Volume License Windows 2012R2 DVD has 4 install images, standard core, standard with GUI, datacenter core, and datacenter with GUI. As we usually build GUI based boxes, we’re only interested in editing those images for now. Launching a PowerShell prompt with elevated access, we need to list the contents of the install iamge:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
F:\Build>dism /Get-ImageInfo /ImageFile:.\Windows\Sources\install.wim

Deployment Image Servicing and Management tool
Version: 6.3.9600.16384

Details for image : .\Windows\Sources\install.wim

Index : 1
Name : Windows Server 2012 R2 SERVERSTANDARDCORE
Description : Windows Server 2012 R2 SERVERSTANDARDCORE
Size : 6,674,506,847 bytes

Index : 2
Name : Windows Server 2012 R2 SERVERSTANDARD
Description : Windows Server 2012 R2 SERVERSTANDARD
Size : 11,831,211,505 bytes

Index : 3
Name : Windows Server 2012 R2 SERVERDATACENTERCORE
Description : Windows Server 2012 R2 SERVERDATACENTERCORE
Size : 6,673,026,597 bytes

Index : 4
Name : Windows Server 2012 R2 SERVERDATACENTER
Description : Windows Server 2012 R2 SERVERDATACENTER
Size : 11,820,847,585 bytes

The operation completed successfully.

As you can see, we have 4 images here, 2 are core (1 and 3) so we’ll ignore those and just work on editing the images we need.

1
2
3
4
5
6
7
F:\Build>dism /Mount-Image /ImageFile:.\Windows\Sources\install.wim /MountDir:.\ISO /Index:2

Deployment Image Service and Management Tool
Version: 6.3.9600.16384

Mounting Image
[==================52.0%               ]

This bit can take a few minutes. Once mounted, if you open Windows explorer to F:\Build\ISO you’ll see a full drive map of an installed Windows machine. This is where we’re going to inject drivers.

1
F:\Build>dism /Image:.\ISO /Add-Driver:.\Drivers /recurse

After a few minutes, you’ll get a nice report of the drivers being added. If you have any unsigned drivers, you can add /ForceUnsigned to the end to make it skip signature validation2.

Now we save the updated image, and close it out.

1
F:\Build>dism /Unmount-Image /MountDir:.\ISO /commit

The /commit forces it to save the changes. If you don’t want to save, use /discard.

We repeated the same steps above, but changed /Index:2 to /Index:4 to mount the datacenter edition of Windows.

Adding Features to the Install Image

One of the other things that we needed to do so we could save a step later was enable features in the new build. Again, dism can handle this by toggling the flag and enabling the features. We wanted MPIO enabled because the B200 has 2 paths due to the chassis they are connected in.

1
2
3
F:\Build>dism /Mount-Image /ImageFile:.\Windows\Sources\install.wim /MountDir:.\ISO /Index:2
F:\Build>dism /Image:.\ISO /Enable-Feature /FeatureName:MultipathIo
F:\Build>dism /Unmount-Image /MountDir:.\ISO /commit

Technically you can save some time by enabling the features at the same time you are injecting the drivers. I’ve just got them separated here.

Adding drivers to Setup and WinPE Images

Adding the drivers to the install image isn’t the end of it. If you’re working with hardware that’s not supported out the box (like the Cisco VICs), then you need to add the drivers to the Setup and WinPE images to make sure they can both see the drives that may be presented from another source (SAN for example). The steps are identical to above, except the image we’re targetting:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
F:\Build>dism /Get-ImageInfo /ImageFile:.\Windows\Sources\boot.wim

Deployment Image Servicing and Management tool
Version: 6.3.9600.16384

Details for image : .\Windows\Sources\boot.wim

Index : 1
Name : Microsoft Windows PE (x64)
Description : Microsoft Windows PE (x64)
Size : 1,321,549,982 bytes

Index : 2
Name : Microsoft Windows Setup (x64)
Description : Microsoft Windows Setup (x64)
Size : 1,417,514,940 bytes

The operation completed successfully.

F:\Build>dism /Mount-Image /ImageFile:.\Windows\Sources\boot.wim /MountDir:.\ISO /Index:1
F:\Build>dism /Image:.\ISO /Add-Driver:.\Drivers /recurse
F:\Build>dism /Unmount-Image /MountDir:.\ISO /commit
F:\Build>dism /Mount-Image /ImageFile:.\Windows\Sources\boot.wim /MountDir:.\ISO /Index:2
F:\Build>dism /Image:.\ISO /Add-Driver:.\Drivers /recurse
F:\Build>dism /Unmount-Image /MountDir:.\ISO /commit

Now our setup image can see the devices that may be required for disk access.

Building the ISO Image

The final step is to turn all this hardwork into a useable ISO/DVD. This is where the ADK comes into play. You’ll need to launch an elevated command prompt using the “Deployment and Imaging Tools Environment” prompt. This sets the %PATH% variables to include some additional tools. We then navigate back to our build directory and start the build process.

1
2
3
C:\>F:
F:\>cd Build
F:\Build>oscdimg -u2 -bf:\build\windows\boot\etfsboot.com f:\build\windows f:\build\win2012r2_b200_20150312.iso

This take a few minutes as it’s making a new ISO. The -u2 argument is used to force UDF file system. This is needed otherwise the install.wim and some other items get trashed by sizing limitations.

Once you have an ISO file, you can either use your favourite ISO burning utility to put it on DVD, or use your servers KVM/ILO/DRAC to remotely mount it to do the install.

All in all, the process takes about 30 minutes depending on the speed of your machine, disks, and drivers/features being enabled. Sadly it took me nearly 2 days to actually build the final image because I had issues identying and including the right chipset drivers.


  1. These instructions work for Vista or higher, so if you’re a 2008 shop, it should work there too.

  2. We found there were several drivers on the Cisco B200 DVD that were unsigned, but they were not needed for our install, so we could skip this.

vSphere Storage vMotion Times Out at 32% When Crossing SANs

A year or so ago we’d upgraded our vCenter from 4.1 to 5.1, and with this upgrade, and some features built into our SAN, we got access VAAI. For example, removing a VM guest would tell the SAN that the guest had been removed, and if the data store had been thinly provisioned from the SAN, it’d clean up and shrink down the space used (in theory).

Another feature we discovered was something called “fast copy”. In layman’s understanding of this feature, when a storage vMotion request was created, the SAN was notified of the request, and the SAN would process the copying of the bits in the background. This is handy because it stops the data from being sent from SAN to host to SAN again. This causes a good speed up with regards to moving machines around.

There was a caveat to the “fast copy” feature that we stumbled across last year. Well, what we stumbled upon was an issue when using vMotion to move machines between SANs. What we didn’t clue in on was that this was because of VAAI and “fast copy”. When we first observed this issue, we didn’t realize the issue was between SANs, we just thought the issue was random. Our VM hosts had storage allocated from 2 different SANs at the time, and our naming convention was a little off, so identifying quickly that the data store was on a different SAN wasn’t entirely obvious at first.

Ultimately the issue presents itself as a vMotion timeout. When you start the vMotion, it zips along until it hits 32%. It then sits there for a few minutes, sometimes up to 5 or 10, then the guest becomes unresponsive. At this point VMware decides the migration has timed out, and rolls back. Sometimes it can take several minutes for the failed guest to start responding again. If the guest is shut down, it usually hangs around 36% for a few minutes, but eventually processes. The error usually looks like this:

SAN vMotion Timeout

The error generally presented is “Timed out waiting for migration data.” It always happened at 32%. A bit of searching around, and I didn’t really uncover the cause of it. At the time we originally spotted this issue, we decided to take an outage and shut the guests down and vMotion them. This killed 2 stones at once, freed memory on the hosts, and gave the guests a reboot to clear memory and such.

Fast forward to nine months ago, and we had an issue where we discovered one of our SANs had become over saturated, and needed space and load removed from it. At this point, we now had a third SAN added to the mix, so we presented new data stores, and went through the process of trying to vMotion quite a lot of VM guests off of one set of data stores (actually 10) to another set. We hit the same wall as before, time outs at 32%. We put it down to the load and space issues on the SAN and went with the outage. This was our dev environment anyway, so it was less of an issue. We didn’t really look into it any further.

Jump forward to this past Tuesday. A sudden alert that multiple VMs had gone offline left us puzzled until we realized that one of the data stores had been way overprovisioned, and the backup software kicked off and with guest snapshots had filled the drive. With a quick bit of work, we moved some guests around, and bumped into the same 32% issue again. Shutting down some guests and shuffling them around got us through the pinch, but left me wondering.

After some experimentation, I was able to narrow down the cause of the issue on a single action. Storage vMotion between SANs. Inner SAN vMotion was snappy, 100GB in less than 2 minutes. Intra-SAN migrations would hit 32% and time out. That’s it, I had the cause of my problem. It had to be a fiber or switch issue… Right?

Not so much. While doing some digging on performance, our fiber switches, and SAN ports, I wasn’t spotting any obvious issues. Doing some searching again on our favourite web search engine, I stumbled across an HP document tucked away in the 3Par area (document was named mmr_kc-0107991, nice name). Bingo! Okay, the details don’t exactly match, for example the document mentions that it freezes at 10%, but it had all the hallmarks of what we were seeing. IntraSAN vMotion, timeouts, and VAAI.

So the solution was to disable VAAI on the host, do the vMotion, and then re-enable it if you still want to use it. VMware has a nice document on how to do that here in KB1033665. With a little PowerCLI1 we quickly disable VAAI and tested a vMotion on a live machine, and it worked. As we were working on a single cluster at the time, this is what we ended up with:

1
2
3
4
5
Get-VMHost -Location (Get-Cluster 'CVHPVMH003') | %{
  Set-VMHostAdvancedConfiguration -VMHost $_ -Name DataMover.HardwareAcceleratedMove -Value 0
  Set-VMHostAdvancedConfiguration -VMHost $_ -Name DataMover.HardwareAcceleratedInit -Value 0
  Set-VMHostAdvancedConfiguration -VMHost $_ -Name VMFS3.HardwareAcceleratedLocking -Value 0
}

Once done, flip the 0s to 1s, and re-enable as needed.


  1. This is something they actually give you in the KB article as well.

Enable-RemoteMailbox - the Address Is Invalid

In the process of migrating our mailboxes from our on-premise Exchange servers to Office 365, we had to rewrite the mailbox enable scripts. This script keys off of our HR database, does some magic, then calls Enable-Mailbox on Exchange 2010 servers. To update this to support creating mailboxes in Office 365, we needed to set user licenses, and use the Enable-RemoteMailbox command in Exchange 20131.

One of the quirks we stumbled upon is a bug in the Exchange 2013 tools that didn’t allow it to identify the domain for the remote routing address. This is what we’d get:

1
2
3
4
5
6
[PS] C:\ > Enable-RemoteMailbox Jonathan.Angliss

The address '@mytenant.mail.onmicrosoft.com' is invalid: "@mytenant.mail.onmicrosoft.com" isn't a valid SMTP address. The domain name can't contain spaces and it has to have a prefix and a suffix, such as example.com.
    + CategoryInfo          : NotSpecified: (:) [Enable-RemoteMailbox], DataValidationException
    + FullyQualifiedErrorId : [Server=Exchsvr01,RequestId=190c9764-d8bd-446e-ac43-7c80bcc54eea,TimeStamp=6/3/2014 1:19:33 PM] [FailureCategory=Cmdlet-DataValidationException] 730D5E7F,Microsoft.Exchange.Management.RecipientTasks.EnableRemoteMailbox
    + PSComputerName        : Exchsvr01

According to the Microsoft documentation for Enable-RemoteMailbox you should be able to specify just the sAMAccountName as an argument, the rest should be calculated.

The remote routing address doesn’t need to be specified because mail flow between the on-premises organization and the service has been configured. Using this configuration, the Enable-RemoteMailbox cmdlet automatically calculates the SMTP address of the mailbox to be used with the RemoteRoutingAddress parameter.

This apparently isn’t the case, so some tweaking was needed. We called upon Get-ADUser to retreive the account, and fill in the rest.

1
[PS] C:\ > Get-ADUser jonathan.angliss | Enable-RemoteMailbox $_.sAMAccountName -RemoteRoutingAddress "$($_.sAMAccountName)@mytenant.mail.onmicrosoft.com"

As this is part of a script, the content is slightly different, but you can see how it works. We used Get-ADUser earlier in the script to pull other user data to calculate licensing requirements, but if you’re doing this as a one off and are seeing the error then you could just as easily do this:

1
[PS] C:\ > Enable-RemoteMailbox jonathan.angliss -RemoteRoutingAddress 'jonathan.angliss@mytenant.mail.onmicrosoft.com'

Hat tip goes to Steve Goodman for posting similar work, and getting me back on track.


  1. If you are using a 2010 Exchange environment, you need a 2013 server to act as a Hybrid server to migrate users.

Exchange 2010, 2013, and Office365: Dynamic Distribution List Filters

In our transition to using Offce365 for email services, we’ve had some interesting discoveries. Some of them are revolving around Dynamic Distribution Lists (DDLs). These are groups which have members identified at time of delivery of emails, and are based on various styles of queries. We usually use PowerShell style queries to build the groups, but LDAP works, as does simple queries based on fixed parameters.

One of the interesting observations is that Exchange will tack extra query parameters into the DDL to exclude system mailboxes. For example the following query string:

1
(RecipientType -eq 'UserMailbox') -and (CustomAttribute4 -eq 'Plano')

Will actually result in the follow query:

1
2
3
4
((((RecipientType -eq 'UserMailbox') -and (CustomAttribute4 -eq 'Plano')) -and (-not(Name -like 'System{*'))
-and (-not(Name -like 'CAS_{*')) -and )-not(-not(RecintTypeDetailsValue -eq 'MailboxPlan'))
-and (-not(RecipientTypeDetailsValue -eq 'DiscoveryMailbox'))
-and (-not(RecipientTypeDetailsValue -eq 'ArbitrationMailbox')))

This forces Exchange to exclude any of the system mailboxes for delivery, which is what you want it to do. The problem is, this additional data varies from version to version, and it’s not always backwards compatible. One of the observations is that in Exchange 2013 they introduced a RecipientTypeDetailsValue of PublicFolderMailbox. This is great, except that value is invalid in 2010. What does that mean?

Let’s try an example. From one of our on-prem 2013 hyrbid servers, we’re going to create a new distribution group with the initial query we gave as an example above…

1
New-DynamicDistributionGroup -Name 'JATestDist' -RecipientFilter "(RecipientType -eq 'UserMailbox') -and (CustomAttribute4 -eq 'Plano')"

Now it’s created, lets see what we have for a query parameter:

1
2
3
4
5
6
7
8
PS C:\> Get-DynamicDistributionGroup -Identity 'JATestDist' | Select RecipientFilter | fl


RecipientFilter : ((((RecipientType -eq 'UserMailbox') -and (CustomAttribute4 -eq 'Plano'))) -and (-not(Name -like
                  'SystemMailbox{*')) -and (-not(Name -like 'CAS_{*')) -and (-not(RecipientTypeDetailsValue -eq
                  'MailboxPlan')) -and (-not(RecipientTypeDetailsValue -eq 'DiscoveryMailbox')) -and
                  (-not(RecipientTypeDetailsValue -eq 'PublicFolderMailbox')) -and (-not(RecipientTypeDetailsValue -eq
                  'ArbitrationMailbox')))

As we can see, a whole bunch of extra options, including our PublicFolderMailbox option. Lets test to see what users we get back…

1
2
3
4
5
6
7
8
[PS] C:\>$dist1 = Get-DynamicDistributionGroup -Identity 'JATestDist'
[PS] C:\>Get-Recipient -RecipientPreviewFilter $dist1.RecipientFilter
$dist1 = Get-DynamicDistributionGroup -Identity 'JATestDist'
Get-Recipient -RecipientPreviewFilter $dist1.RecipientFilter

Name                                                        RecipientType
----                                                        -------------
Angliss, Jon                                                UserMailbox

Okay, so we get results back, no big deal right? Now I’m going to go to a 2010 server, and without changing anything, I’m going to see what results we get back…

1
2
3
4
5
[PS] C:\>$dist1 = Get-DynamicDistributionGroup -Identity 'JATestDist'
[PS] C:\>Get-Recipient -RecipientPreviewFilter $dist1.RecipientFilter
The recipient preview filter string "((((RecipientType -eq 'UserMailbox') -and (CustomAttribute4 -eq 'Plano'))) -and (-not(Name -like 'SystemMailbox{*')) -and (-not(Name -like 'CAS_{*')) -and (-not(RecipientTypeDetailsValue -eq 'MailboxPlan')) -and (-not(RecipientTypeDetailsValue -eq 'DiscoveryMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'PublicFolderMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'ArbitrationMailbox')))" is neither a valid OPath filter nor a valid LDAP filter. Use the -RecipientPreviewFilter parameter with either a valid OPath filter string or a valid LDAP filter string.
    + CategoryInfo          : InvalidArgument: (:) [Get-Recipient], ArgumentException
    + FullyQualifiedErrorId : 79B22B5B,Microsoft.Exchange.Management.RecipientTasks.GetRecipient

Now we see an error. This is because 2010 doesn’t like PublicFolderMailbox as a RecipientTypeDetailsValue, and as such, throws it out as an error. So we have to go back to the 2010 server and edit the query, and reset it to be what we wanted originally:

1
2
3
4
5
6
7
8
9
[PS] C:\>Set-DynamicDistributionGroup -Identity 'JATestDist' -RecipientFilter "(RecipientType -eq 'UserMailbox') -and (CustomAttribute4 -eq 'Plano')"
[PS] C:\>$dist1 = Get-DynamicDistributionGroup -Identity 'JATestDist'
[PS] C:\>Get-Recipient -RecipientPreviewFilter $dist1.RecipientFilter
$dist1 = Get-DynamicDistributionGroup -Identity 'JATestDist'
Get-Recipient -RecipientPreviewFilter $dist1.RecipientFilter

Name                                                        RecipientType
----                                                        -------------
Angliss, Jon                                                UserMailbox

This same query is happy on the 2013 servers as well, however, it will attempt delivery to a Public Mailbox if your other query parameters allow for it. In the example above, we set the RecipientType to be a very specific value, so this shouldn’t allow for this to happen anyway.

One other observation, when migrating your queries to be Office 365 Hybrid complaint, you will also need to include the RecipientType of MailUser. For example:

1
((RecipientType -eq 'UserMailbox') -or (RecipientType -eq 'MailUser')) -and (CustomAttribute4 -eq 'Plano')

Mailboxes that are migrated change their RecipientType to be MailUser.

There are lots of other fun things about DDLs that you’ll have to be aware of, which I shall cover in a separate post, but this is one of the fun gotchas I discovered in a mixed environment that’ll impact people using Exchange 2010 and 2013 in the same environment.

Exchange and the Case of the Missing Counters

While setting up SolarWinds SAM AppInsight for Exchange, I stumbled across a small Exchange setup bug where it’s not correctly deploying all the counters for the server roles that are being used. When SAM does the checks for the performance counter, you’ll see an error like the following:

1
2
'Average Document Indexing Time'
'Performance counter not found'

The solution is fairly simple, you have to copy the counters from the install media, and register them using PowerShell. The one caveat is that the counters aren’t on all the install media, so if you have CU3 setup files for example, they are not there. You have to go back to the original install DVD and get them from there. Here are the steps:

  1. Find the missing performance counter files on the install media, usually in <install media>\setup\perf
  2. For the above counter, the files needed are:
    1. IndexAgentPerfCounters.h
    2. IndexAgentPerfCounters.ini
    3. IndexAgentPerfCounters.xml
  3. Copy the files to <install path>\Setup\perf
  4. Open a PowerShell prompt with elevated privileges
  5. Execute the following commands:
1
2
Add-PSSnapin Microsoft.Exchange.Management.PowerShell.Setup
New-PerfCounters -DefinitionFileName "<install path>\Setup\Perf\IndexAgentPerfCounters.xml"

Obviously adjust <install path> to be the correct path.

This is documented on the SAM product blog (here) and in the SolarWinds Knowledge Base (here), with the missing step of Add-PSSnapin.

L vs R and the Importance of Documentation

This post was going to be one of those rant posts about not following instructions, and then I realized this is a common problem that a lot of people have issues with, not just in the IT world. It is the importance of knowing what L and R is referring to. By L and R, I mean left and right.

This might seem silly and trivial, because we all know our left hand and our right hand, and in general when somebody says it’s in the left hand side of the cabinet, you know exactly where to look. But what happens if you are working on something that can be reached/used from both sides? Is that the left hand side relative to the front, or the back?

This comes up with other things too. How many times have you gone to a car mechanic and said “my left brake squeals when I apply the brakes”? Is that the left side when looking at the car, or driving the car? This is why you’ll find a lot of mechanics refer to the sides as driver and passenger, there is no confusion there.

The whole point of this post is about the importance of documenting exactly what is referred to as L and R, because it makes a great deal of difference when you are putting rails on a server. Why you might ask? It’s all about server removal…

A lot of rail kits consist of 3 major parts, the server portion, the cabinet/rack portion, and the slider. The server portion is usually screwed/clipped onto the server, and is stationary. The cabinet/rack portion is also stationary, and attached to the cabinet/rack. Then there is the slider portion. This portion is “attached” to the server and cabinet portions, has ball bearings, and allows the server to slide in and out of the rack. It slides on tracks in the cabinet portion, and the server portion slides on bearings to slide out. This allows for people to work on the server without having to completely remove the server.

Also part of the slider is usually a catch. This catch is to stop you from pulling the server completely off the rails and having it come crashing down to the floor. Something most people don’t want to happen. And it is with this part of the rails that it is important to know what is L and what is R. This catch usually has an orientation so that it can “clip” into the slider rail, and pushing the catch down allows the server rail to slide out of the slider rail. If you mount the server rail on the wrong side, the catch either doesn’t work properly, or becomes impossible to remove.

Here is an example of one of those catches…

Rail Catch mechanism

If looking at this, you cannot figure out how this works, here is another picture with arrows. Arrows make everything easier to understand…

Rail Catch mechanism directions

When you pull the server out, it moves in the direction of the top arrow (orange). Near the end of the slider rail is a small block, this block (shows as a green blob) moves along the server rail in the direction of the bottom arrow (green). As it gets to the catch, it pushes it up, and the spring in the catch pushes it back down when the block moves into the void. Because of the shape of the void, the green blob is prohibited from moving any further, and stops the server sliding off the end of the rail.

If you need to actually remove the server from the rails completely, you simply pull the catch up, which moves the blob outside the void of the catch, and pull the server forward. If you put the rail on upside down, instead of the block catching on the void in the catch, it actually stops when it hits the mount point of the catch. This is why it’s important to know which way around to mount the rails (note the little L next to the screw).

This situation caused myself and a co-worker some struggles as we could not get the server unmounted from the rails. Ultimately we ended up having to unscrew the rails from the rack, with the server still attached, fully extend the rails, and then bend them in so that we could pull the server out of the rack. Fortunately this was a sever that is well past EoL, so this wasn’t a hard decision to make, or live with.

Server rail bend 1

Server rail bend 2

That all being said, it is important to make documentation as clear and concise as possible. Images are very useful in this situation. A server we put in place of this one had really clear documentation, and the rails themselves even had pictures of the configuration, essentially saying “This rail goes on the left here” with a picture of where the rail was located in relation to the server.

So next time you’re writing documentation for something, and there is an opportunity for ambiguity, clear up the documentation and remove any doubt.

Unable to Remove Exchange Mailbox Database

We had an odd issue recently where our Exchange server refused to let us remove a mailbox database, citing that the database had one or more mailboxes. The exact error was this:

This mailbox database contains one or more mailboxes, mailbox plans, archive mailboxes, public folder mailboxes or arbitration mailboxes. To get a list of all mailboxes in this database, run the command Get-Mailbox -Database . To get a list of all mailbox plans in this database, run the command Get-MailboxPlan. To get a list of archive mailboxes in this database, run the command Get-Mailbox -Database -Archive. To get a list of all public folder mailboxes in this database, run the command Get-Mailbox -Database -PublicFolder. To get a list of all arbitration mailboxes in this database, run the command Get-Mailbox -Database -Arbitration. To disable a non-arbitration mailbox so that you can delete the mailbox database, run the command Disable-Mailbox . To disable an archive mailbox so you can delete the mailbox database, run the command Disable-Mailbox -Archive. To disable a public folder mailbox so that you can delete the mailbox database, run the command Disable-Mailbox -PublicFolder. Arbitration mailboxes should be moved to another server; to do this, run the command New-MoveRequest . If this is the last server in the organization, run the command Disable-Mailbox -Arbitration -DisableLastArbitrationMailboxAllowed to disable the arbitration mailbox. Mailbox plans should be moved to another server; to do this, run the command Set-MailboxPlan -Database .

Okay, so thinking we were being stupid and missed the arbitration mailboxes, we ran the recommended commands, with no such luck:

1
2
[PS] D:\>get-mailbox -database 'Mailbox Database 2102391437' -Arbitration
[PS] D:\>

The same was true of mailbox plans, and archive mailboxes. After some head scratching, I stumbled across this post on TechNet. The basic gist is that because Exchange is in a multi-domain forest, the get-mailbox command will usually only search in the domain you are active in. To make Exchange operate outside of the working domain, you have to set the server settings.

1
2
3
4
5
6
7
8
[PS] D:\>set-adserversettings -ViewEntireForest $true
[PS] D:\>get-mailbox -database 'Mailbox Database 2102391437'
[PS] D:\>get-mailbox -database 'Mailbox Database 2102391437' -Arbitration

Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
SystemMailbox{bb558c35... SystemMailbox{bb5... msg014a          Unlimited
Migration.8f3e7716-201... Migration.8f3e771... msg014a          300 MB (314,572,800 bytes)

Sure enough, those system mailboxes hiding out in the mailbox database. Now we can see them, we can move the mailboxes off of the database, and then remove the database.

SolarWinds Application Monitor - Automatically Configuring AppInsight for Exchange

I’m going to make some assumptions in this post as it’s about a specific product. First, let us assume you are a long time user of SolarWinds Server & Application Monitor (previously known as Application Performance Monitor). Let’s also assume you have a Microsoft Exchange 2010 (or 2013) environment. I’m also going to assume that you have WMI monitoring working for your servers. And for the final assumption, let’s say that you just found out that the latest SAM update (6.1) now includes Exchange monitoring ‘out of the box’. After you’ve finished doing a happy dance, and that you no longer have to tinker with all the extra application monitors now, you set about figuring out how to enable this new functionality.

First, there are some caveats. The first is that this new functionality is only targeted for the Exchange Mailbox role. This means that if you have separated roles, such as CAS or Transport, don’t bother trying to point it at those servers, it just won’t find anything1.

The second caveat is permissions. To let the auto-configuration work (which is what this post will be about), you’ll need to have the account SAM uses have temporary administrative access to the mailbox server.

Now you’ve added your WMI service account to the “Administrators” group on your Mailbox servers. The next step is to make sure your service account has the right access within Exchange. There are 2 roles the account needs, Mailbox Search and View-Only Organization Management. The latter can be handled by adding the service account to the role that is already defined. The former needs to be created specifically for this purpose.

Now let’s see what we have to do in SAM. There are 2 ways to do this, I’m going with the way I’m familiar with, and it’s the same as if you add new drives/volumes, or extra hardware to a server. Locate your server in the SAM interface, scroll down to the “Management” box, and click on “List Resources”. The other method is to use the Sonar Discovery tool.

SAM Node Management

Let SAM do its work, and wait patiently. This can take a minute or two depending on the load on both servers. Once it has finished its autodiscovery process, you should now see new applications under the AppInsight Applications umbrella, check the box and click “Save”

SAM Node Resources

Once you’ve done this, your “Application Health Overview” section should now show an application in an “Unknown” status.

SAM App Health

Click on the “Unknown” line and you’ll be taken to a view listing the unknown applications. This should (hopefully if you’ve set stuff up right) be just the Microsoft Exchange app. Click on that.

SAM Unknown Apps

At the top of the page, in the “Management” section, click on the “Edit Application”. There are 3 important items on the page that follows. The first is the URLs, if you use the defaults for most of your environment for configuration, these should probably be left alone. The Windows URL for PowerShell is for the remoting functionality, which will be configured automatically if you have not already done so. The next is the server credentials used to access Exchange and the server. Usually the “Inherit Windows credential from node” is good enough, assuming the monitoring service account is the same you want to use for monitoring.

SAM Exchange URLs

Now we’ve got this far, the last thing to do is hit the “Configure Server” button. This process configures WinRM, and the Exchange PowerShell web services for SAM to access.

SAM Configure Exchange Server

Usually this step can take a minute or two, and is a perfect time to go grab yourself a drink. When you return, and everything says it was successful, hit the “Test Connection” button just to make sure.

If you’re curious about what goes on behind the scenes of the “Configure Server” button, I’ll also be writing up the manual process, which is exactly what this does.

You are now ready to enjoy statistics and monitoring for your Exchange Mailbox server, including items such as largest mailbox, quota usage, messages sent, DAG and cluster statuses, and the likes.

Exchange - Users by Messages Sent Exchange - Users by Mailbox Size Exchange - Database Size and Space Use Exchange - Database Copies

So far I’m very happy about getting this deployed. It’s actually giving me some numbers behind some of the concerns I’ve had since I started where I work. For example, we have quotas that restrict sending, but not delivery, so mailboxes can exceed the quota size by a lot, as long as they are receiving only.

Edit (04/22/2014): It appears that when answering a question on Thwack, part of the problem I had locating the automatic configuration is that there may have been a step missing. The documentation mentions to go to the “All Applications” resource, which is not on the node page, but on the Applications / SAM Summary page. The thread with that conversation on is here. As of 13:57 US Central time, they have added that the documentation will be updated to clear up the confusion.


  1. This gives me a sad face, and I’m hoping that it’ll be added as new features in upcoming releases.

Azure VMs and Setting Subnets via PowerShell

One of the projects I’ve been working on recently is a POC in Azure to allow us to move a collection of desktop users to lower end laptops, while using high end servers to perform a lot of data processing. The idea is that we can spin up and destroy machines as we see fit. The plan was fairly solid, and we build out our domain controllers and a template machine with all the software in it, before configuration. We then used PowerShell to spin up new machines as we needed them.

One of the issues I stumped over when working on this was making sure the servers were put into the right network. This was important as they were being joined to a domain. I had originally started with something like this:

1
2
3
4
5
6
7
8
9
$img = 'imgid_Windows-Server-2008-127GB.vhd'
$svcname = 'mytestservice01'
$svcpass = '!testpass321!'
$svcuser = 'testadmin'

$vm1 = New-AzureVMConfig -ImageName $img -InstanceSize 'ExtraSmall' -Name $svcname | `
  Add-AzureProvisioningConfig -WindowsDomain -AdminUsername $svcuser -Password $svcpass -DomainUserName 'dmnadmin' -Domain 'TestDomain' -DomainPassword 'ImnotTelling!' -JoinDomain 'TestDomain.local' -TimeZone 'Canada Central Standard Time'

New-AzureVM -VMs $vm1 -ServiceName $svcname -VNetName 'Test_Net' -AffinityGroup 'TestGroup-USEast'

This seemed to look right, and worked fine, as long as I wasn’t trying to add it to a VNet or an Affinity Group. When I added those options, I was thrown the following error:

1
New-AzureVM : Networking.DeploymentVNetAddressAllocationFailure : Unable to allocate the required address spaces for the deployment in a new or predefined subnet that is contained within the specified virtual network.

It seemed to me that the New-AzureVM command should have had some method to define which subnet was to be allocated to, but it wasn’t there. What was even more confusing was this VNet only had a single subnet, so you’d think it might select that, but not so much luck.

The answer lies in the Set-AzureSubnet command, which should have been pretty obvious to me. You can add it as part of your provisioning command like this:

1
2
3
$vm1 = New-AzureVMConfig -ImageName $img -InstanceSize 'ExtraSmall' -Name $svcname | `
    Add-AzureProvisioningConfig -WindowsDomain -AdminUsername $svcuser -Password $svcpass -DomainUserName 'dmnadmin' -Domain 'TestDomain' -DomainPassword 'ImnotTelling!' -JoinDomain 'TestDomain.local' -TimeZone 'Canada Central Standard Time' | `
  Set-AzureSubnet 'Subnet-1'

All I’ve done is added the extra command to the end, and now Azure is happy. This will spin up a new VM and drop it in the right VNet, Affinity Group, and Subnet. Based on the VNet’s network configurations, and DNS settings, the new machine is provisioned, and joined to the domain immediately.

This makes me very happy because this is a quick sample of how we’d proceed with automating and deploying an undefined number of VMs in Azure based off of our golden image. With some minor tweaks we can loop through and spin up 50 machines with little work.