Operational Readiness validation gotchas

Sup’ PSHomies,

Back from the PSConfEu 2016 in Hannover! It was awesome!!! It was great meeting so many in person! I highly recommend attending a conference if ever given the chance! Tobias did a great job organizing PSConfEU 2016!

The presentations were top notch! Two presentations I definitely wanted to follow were June Blenders’ Real world Pester TDD tests & Ravikanth Chaganti Operations Validation Framework. I enjoyed Ravikanth’s approach: a simplistic and a comprehensive test. June’s presentation was insightful! There were definitely a few aha moments for me! So I decided to re-evaluate what I had learned so far now that I’ve seen how it should be done! 😉 .

Simplistic tests

Think of simplistic tests as kicking the tires. Here’s where the obvious tests go:

  • Are the sets aligned?
  • Did the object count meet your expectation?

Things of that nature. Here are some tests to help you understand some gotcha moments.

Validating the count seems pretty straight forward right? Well, not always… I’ll explain…

When the sets to validate are identical validation is pretty straight forward. This is the best case scenario. I did one validation with and without Group-Object (I’ll explain later on).

Simplistic Identical sets - Count

Simplistic Indentical sets

By using Group-Object I can get the “real” count of a set. Group-Object will gauge the uniqueness of the set, but I also found a test where that might not always be useful… So my next test was to omit an entry from the Verify set

Simplistic Missing entry sets - Count

Simplistic missing an entry in Verify

The test failed as it should. Now for some fun, let’s add a double entry to the Verify set.

 

Simplistic Double entry sets - CountThis is an odd test, it could be a typo. Imagine having a list where double entries aren’t that obvious, this should catch it. Now ideally you’d use your source code that you used during deployment. I’m assuming you did automate your process eh? 😉 Without the Group-Object the count is identical. With Group-Object you only have one entry! This could explain an exception happened during deployment… Accidents will happen…

The last test is a fun one: different sets.

Simplistic Different sets - Count

Different sets

Surprise! Both validation count test passed, but the sets are totally different!

Bonus Test!

Simplistic Different sets double entry - Count

Different set with a double entry

This one almost got away. While doing the comprehensive test, it dawned on me that I should be testing both counts, with and without Group-Object. I updated the screenshot accordingly.

Take away Simplistic test:

Don’t only rely the count of a set. By using Group-Object you can gauge a set’s uniqueness. The bonus test showed that exceptions may happen. It’s totally valid as a starting point, that’s why it’s a simplistic tests. Here’s the code for the Simplistic test for count validation:

$savedADConfig= @{
   GlobalCatalogs = @(
      'DC-DSC-01.pshirwin.local'
      'DC-DSC-02.pshirwin.local'
   )
}

$verifyADConfig= @{
   GlobalCatalogs = @(
      'DC-DSC-01.pshirwin.local'
      'DC-DSC-02.pshirwin.local'
   )
}

#region Example Operational validation Simplistic test
Describe 'Active Directory configuration operational readiness' {
   Context 'Verifying GlobalCatalogs count without Group-Object'{
      it 'Total GlobalCatalogs match' {
         @($savedADConfig.GlobalCatalogs).Count |
         Should be @($verifyADConfig.GlobalCatalogs).Count
      }
   }

   Context 'Verifying GlobalCatalogs count with Group-Object'{
      it 'Total GlobalCatalogs match' {
         @($savedADConfig.GlobalCatalogs  | Group-Object).Count |
         Should be @($verifyADConfig.GlobalCatalogs | Group-Object).Count
      }
   }
}
#endregion

If you’d like to try out the simplistic tests just add/remove entries to the saved-/verifyADConfig sets. Now for the Comprehensive tests!

Comprehensive tests

Here’s where in-depth analysis goes. When I did the AD Operational Readiness test, I had a feeling I was missing something. I saw June using sort-object in one of her validations. That triggered me to re-valuate this test.

Comprehensive Identical sets

Comprehensive Identical sets

Depending on which set you used for your enumeration you could end up with different results. When the sets are identical, all goes well. Next test, omit an entry in Verify set.

Comprehensive Missing an entry

Comprehensive missing an entry

Enumerating from the saved test caught the missing entry, enumerating from verify didn’t. Both found ‘DC-DSC-01.pshirwin.local’. The simplistic test caught this, that is why you need both! Next up: Double entry in Verify.

Comprehensive Double entry

Comprehensive double entry in verify

Enumerating from the saved set caught the double entry. Enumerating from the verify set just enumerated the entry twice. If you’re visually inclined, you might miss this.

Hey all my tests results are green and purple! Yeah…

Last test: Different sets.

Comprehensive Different sets

Different sets

At this point you’re comparing apple with oranges. This should fail.

Take away comprehensive tests:

The set you’re enumerating from matters! To cover validation, best bet is to do both! Here’s the code for the Comprehensive Test:

#region Example Operational validation Comprehensive test
$savedADConfig= @{
   GlobalCatalogs = @(
      'DC-DSC-01.pshirwin.local'
      'DC-DSC-02.pshirwin.local'
   )
}

$verifyADConfig= @{
   GlobalCatalogs = @(
      'DC-DSC-01.pshirwin.local'
      'DC-DSC-02.pshirwin.local'
   )
}

Describe 'Active Directory configuration operational readiness' {
   Context 'Verifying GlobalCatalogs enumerating from saved configuration'{
      $savedADConfig.GlobalCatalogs |
      ForEach-Object{
         it "Server $($_) is a GlobalCatalog"{
            $verifyADConfig.GlobalCatalogs.Contains($_) |
            Should be $true
         }
      }
   }
   Context 'Verifying GlobalCatalogs enumerating from verify configuration'{
      $verifyADConfig.GlobalCatalogs |
      ForEach-Object{
         it "Server $($_) is a GlobalCatalog"{
            $savedADConfig.GlobalCatalogs.Contains($_) |
            Should be $true
         }
      }
   }
}
#endregion

Validation is great, but you need to make sure your validating with the right set in the right order!

Summary

  • Create Simplistic & Comprehensive tests.
  • Simplistic tests should take care of the obvious.
  • Comprehensive tests is where in-depth analysis takes place.
  • Validate by enumerating from both sets!
  • Never trust a test that doesn’t fail 😉

I’m glad I visited both presentations! Now it’s time to update my Operational readiness tests accordingly! 🙂

Hope it’s worth something to you

Ttyl,

Urv

Advertisements

6 thoughts on “Operational Readiness validation gotchas

  1. Pingback: How I Work: Pester Testing - PowerShell.Getchell.Org

  2. Mark Wragg

    Hi Irwin,

    I’ve spent most of today building on your original AD Operational Readiness work which was fantastic. This post is really helpful too, it had occurred to me that we should be able to fail a test if an additional DC had been added for example and i’m not sure the existing test would do so.

    I have also added an AD health section, which checks that certain services are running, TCP ports respond as well as calls DCDiag, NLTest, RepAdmin and evaluates their results for errors. I’m thinking of splitting that in to a separate script ultimately as while we might sometimes call them together (and named properly/stored together invoke-pester would anyway) sometimes we might want to just evaluate health and not config.

    I’ve put my work in the PS Gallery and would love your feedback: https://www.powershellgallery.com/packages/ADAudit/1.0

    In particular I have changed it so that the configuration we’re validating against (which i’ve referred to as “gold”) is just kept in XML, rather than embedding it in the script. Any reason why that might be a bad idea?

    I’m going to blog about this and my additions to your work (with full credit to you) in the near future, I hope you don’t mind.

    Best Wishes,
    Mark

    Liked by 1 person

    Reply
    1. Irwin Strachan Post author

      Hi Mark, Thanks for your feedback! The reason for the code being embedded was that I did’t have the source with which it was implemented. Ideally you’d use the source that you used to automate the process 🙂

      I recently discovered tags in Pester. Using Tags will help you run only specific test. Here’s a quick example Invoke-Pester .\ps1\dfsn -Tag DFSnShares -OutputFile .\export\dfsn\DFSnShares.NUnit.xml -OutputFormat NUnitXml -PassThru

      You could create a Tag for DCDiag,RepAdmin test and only run those using the tag option with invoke-pester

      I look forward to reading about it! 🙂

      Rg./Irwin

      Liked by 1 person

      Reply
  3. Pingback: AD Operation Validation class | pshirwin

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s