Test Plan Status Report

Description

The Test Plan Status is a report that extends the functionality found in the Test tab of Microsoft Test Manager.  While Test Manager can show you the completion status of the selected Test Suite, it does not show roll-up information when you have a hierarchy of suites.  This report is intended to fill that gap.

Deployment

Follow the instructions in Documentation > Report Installation to install this report in a team project’s SQL Server Reporting Services site.  This report uses the reporting site path to determine the current team project information.  This report must be installed under each team project that will use it.

Usage

The Test Plan Status report takes only one parameter, the name of a Test Plan. When run, the Test Plan Status report takes the Test Plan Name argument and reads the TFS Data Warehouse to pull in all of the then-current test run data for that plan.  The Test Plan Name list is populated from all of the test plans in the team project. 

Definitions

  • Test Suite – a grouping level within a Test Plan in Microsoft Test Manager. A Test Suite may contain Test Cases or Test Suites.
  • Test Case – a type of work item in Team Foundation Server that describes a test scenario. It usually contains a series of steps and expected outcomes.
  • Test Point – any given test case can be run multiple times under different conditions. The test may be a combinatorial scenario, such as a Login test in which the same test steps are run but differing inputs are used.  Each unique set of inputs requires that the test be run an additional time. Each of these iteration is considered a Test Point.  So if we have a test case that has 3 rows of test input data then that test case is said to have 3 test points.
  • Passing Test – a test that was run and the application under test performed as expected.
  • Failing Test – a test that was run and the application under test did not perform as expected.
  • Blocked Test – a test that was run but could not complete due to a condition outside the control of the tester, such as a needed reference server becoming unavailable during the test run. This is not a test failure since the test could not complete and the problem is not an application issue.
  • Never Run Test – a test that has never been run within the current Test Suite.  It is possible for a Test Case to exist in multiple test suites in the same test plan.  If that test has been run in one suite, that run is not credited to the other suite.  The tester must run that test in the context of each suite it participates in to avoid being counted as Never Run.  Once a test has been run once in a suite, it can never o back to the Never Run state.  It changes to the Active state.
  • Active Test – This is a test that has been run in the past and has either passed, failed or been blocked.  If it is reset, it moves to an Active state. 
    For example: You have run a test during a development cycle and that test has passed.  You then find out that the development team has made some code changes to that area of the application thus invalidating your earlier test results.  You can change the passed tests back to Active to denote that they need to be run again.

Interpretation

NOTE: The report will only show a Test Suite if that Suite or one of its children contains at least one Test Case.  Empty Test Suites do not display in the report. This is similar behavior to the Test Activity within Microsoft Test Manager. 

The report displays the following data for each Test Suite:

  • Test Suite Name (indented to show level of hierarchy)
  • Number of test points passing
  • Number of test points failing
  • Number of test points in a Blocked state
  • Number of test points that have never been run
  • Number of test points that have been previously run but have been reset to an Active state by the user
  • A bar chart showing the above test points breakdown
  • The date and time of the most recent run of any test in that suite.

 

SNAGHTML8efdec8

No Data Available for Report

When there is no data available based on the Test Plan selected the report will show a message stating:

No data was found for this Test Plan.  Test Suites will only appear in this view if the Suite contains one or more Test Cases. Test Suites will also appear if one of their child Suites contains one or more Test Cases.

This will be the case if there are no test cases in any of the test suites in the selected plan.  Adding a single test case anywhere in the plan (and waiting for the warehouse to be refreshed) will remove this message.

image

Last edited Aug 19, 2012 at 9:31 PM by sstjean, version 9

Comments

erint Nov 7, 2013 at 11:10 AM 
We have tested the Test Plan Status report (version 1.0.0.stable), and find that it is pulling data from other test plans for other team projects from different team project collections. We have an example where the report pulls data from 26 additional test plans (from 3 other team projects in 3 different collections) in addition to the selected one. This quickly makes the report unreadable. It also breaks with the security principle that team reports show data at the team project level. Is there a parameter I can configure to get the report to only pull data for the selected team project test plan?

KLyovin Oct 14, 2013 at 1:24 PM 
As well as Query-based suites. Only obvious hierarchic test suites are supported.

KLyovin Oct 14, 2013 at 1:21 PM 
It seems like it doesn't currently support Requirement-based Suites.