Operational Readiness validation gotchas

Sup’ PSHomies,

Back from the PSConfEu 2016 in Hannover! It was awesome!!! It was great meeting so many in person! I highly recommend attending a conference if ever given the chance! Tobias did a great job organizing PSConfEU 2016!

The presentations were top notch! Two presentations I definitely wanted to follow were June Blenders’ Real world Pester TDD tests & Ravikanth Chaganti Operations Validation Framework. I enjoyed Ravikanth’s approach: a simplistic and a comprehensive test. June’s presentation was insightful! There were definitely a few aha moments for me! So I decided to re-evaluate what I had learned so far now that I’ve seen how it should be done! 😉 .

Simplistic tests

Think of simplistic tests as kicking the tires. Here’s where the obvious tests go:

• Are the sets aligned?
• Did the object count meet your expectation?

Things of that nature. Here are some tests to help you understand some gotcha moments.

Validating the count seems pretty straight forward right? Well, not always… I’ll explain…

When the sets to validate are identical validation is pretty straight forward. This is the best case scenario. I did one validation with and without Group-Object (I’ll explain later on).

Simplistic Indentical sets

By using Group-Object I can get the “real” count of a set. Group-Object will gauge the uniqueness of the set, but I also found a test where that might not always be useful… So my next test was to omit an entry from the Verify set

Simplistic missing an entry in Verify

The test failed as it should. Now for some fun, let’s add a double entry to the Verify set.

 

This is an odd test, it could be a typo. Imagine having a list where double entries aren’t that obvious, this should catch it. Now ideally you’d use your source code that you used during deployment. I’m assuming you did automate your process eh? 😉 Without the Group-Object the count is identical. With Group-Object you only have one entry! This could explain an exception happened during deployment… Accidents will happen…

The last test is a fun one: different sets.

Different sets

Surprise! Both validation count test passed, but the sets are totally different!

Bonus Test!

Different set with a double entry

This one almost got away. While doing the comprehensive test, it dawned on me that I should be testing both counts, with and without Group-Object. I updated the screenshot accordingly.

Take away Simplistic test:

Don’t only rely the count of a set. By using Group-Object you can gauge a set’s uniqueness. The bonus test showed that exceptions may happen. It’s totally valid as a starting point, that’s why it’s a simplistic tests. Here’s the code for the Simplistic test for count validation:

$savedADConfig= @{
   GlobalCatalogs = @(
      'DC-DSC-01.pshirwin.local'
      'DC-DSC-02.pshirwin.local'
   )
}

$verifyADConfig= @{
   GlobalCatalogs = @(
      'DC-DSC-01.pshirwin.local'
      'DC-DSC-02.pshirwin.local'
   )
}

#region Example Operational validation Simplistic test
Describe 'Active Directory configuration operational readiness' {
   Context 'Verifying GlobalCatalogs count without Group-Object'{
      it 'Total GlobalCatalogs match' {
         @($savedADConfig.GlobalCatalogs).Count |
         Should be @($verifyADConfig.GlobalCatalogs).Count
      }
   }

   Context 'Verifying GlobalCatalogs count with Group-Object'{
      it 'Total GlobalCatalogs match' {
         @($savedADConfig.GlobalCatalogs  | Group-Object).Count |
         Should be @($verifyADConfig.GlobalCatalogs | Group-Object).Count
      }
   }
}
#endregion

If you’d like to try out the simplistic tests just add/remove entries to the saved-/verifyADConfig sets. Now for the Comprehensive tests!

Comprehensive tests

Here’s where in-depth analysis goes. When I did the AD Operational Readiness test, I had a feeling I was missing something. I saw June using sort-object in one of her validations. That triggered me to re-valuate this test.

Comprehensive Identical sets

Depending on which set you used for your enumeration you could end up with different results. When the sets are identical, all goes well. Next test, omit an entry in Verify set.

Comprehensive missing an entry

Enumerating from the saved test caught the missing entry, enumerating from verify didn’t. Both found ‘DC-DSC-01.pshirwin.local’. The simplistic test caught this, that is why you need both! Next up: Double entry in Verify.

Comprehensive double entry in verify

Enumerating from the saved set caught the double entry. Enumerating from the verify set just enumerated the entry twice. If you’re visually inclined, you might miss this.

Hey all my tests results are green and purple! Yeah…

Last test: Different sets.

Different sets

At this point you’re comparing apple with oranges. This should fail.

Take away comprehensive tests:

The set you’re enumerating from matters! To cover validation, best bet is to do both! Here’s the code for the Comprehensive Test:

#region Example Operational validation Comprehensive test
$savedADConfig= @{
   GlobalCatalogs = @(
      'DC-DSC-01.pshirwin.local'
      'DC-DSC-02.pshirwin.local'
   )
}

$verifyADConfig= @{
   GlobalCatalogs = @(
      'DC-DSC-01.pshirwin.local'
      'DC-DSC-02.pshirwin.local'
   )
}

Describe 'Active Directory configuration operational readiness' {
   Context 'Verifying GlobalCatalogs enumerating from saved configuration'{
      $savedADConfig.GlobalCatalogs |
      ForEach-Object{
         it "Server $($_) is a GlobalCatalog"{
            $verifyADConfig.GlobalCatalogs.Contains($_) |
            Should be $true
         }
      }
   }
   Context 'Verifying GlobalCatalogs enumerating from verify configuration'{
      $verifyADConfig.GlobalCatalogs |
      ForEach-Object{
         it "Server $($_) is a GlobalCatalog"{
            $savedADConfig.GlobalCatalogs.Contains($_) |
            Should be $true
         }
      }
   }
}
#endregion

Validation is great, but you need to make sure your validating with the right set in the right order!

Summary

• Create Simplistic & Comprehensive tests.
• Simplistic tests should take care of the obvious.
• Comprehensive tests is where in-depth analysis takes place.
• Validate by enumerating from both sets!
• Never trust a test that doesn’t fail 😉

I’m glad I visited both presentations! Now it’s time to update my Operational readiness tests accordingly! 🙂

Hope it’s worth something to you

Ttyl,

Urv