Organizing Tests into Suites

Overview

Unless you specify otherwise, stride will publish results from all of your test units into a single top-level test suite when run. This is a satisfactory arrangement as long as the number of tests is relatively small. As the number of test units grows, or test units from disparate groups are aggregated (as in a Continuous Integration arrangement) this flat report organization becomes hard to use.

The software under test itself is typically broken up into separate units (e.g. libraries, components, objects, etc.) and organized into a hierarchy that reflects the functional organization of the software. If the results from STRIDE tests could be organized identically to the software under test, understanding and analyzing test results would be much simpler.

Organizing Into Suites at Test Runtime

The stride test runner allows you to specify a "path" under which a test unit's results will be published. The path is a forward-slash delimited list of suite names. The path is optionally specified in the argument to the -r command line option.

Following are some examples. Many more examples can be seen here:.

<-r omitted>

If you omit the -r argument, stride will run all of the test units specified in the database and publish them into a single top-level root test suite

-r /(TestUnit1, TestUnit2)

Here we specify two test units to be run and published to the root suite

-r /suite1(TestUnit1) -r /suite2(TestUnit2)

Here the two test units are each published to their own suite. (Note that -r can be specified any number of times on a stride command line.)

-r /suite1/suite1.1/suite1.1.1(TestUnit1, TestUnit2)

Here, the two test units are published to a suite that is several levels deep in a hierarchy. Note that stride will create suites as needed to fulfill your publishing requirements.

Another important benefit to be gained by organizing your test units into suites is that a subtotal roll-up of several important results are provided by each suite. Pass/fail counts, error counts, and time under tests are all summed at the suite level for all tests that are under the current suite and any of its subsuites.

Techniques for Test Organization

Using an Options File

One way to go about organizing an aggregation of tests into suites would be to create an options file comprising stride run options that map tests to the suites under which they are to be published. This options file would then be submitted to stride (along with other options).

This technique has a big shortcoming in an environment where test units may be added or removed. Even if controls are put into place, the options file will inevitably get out of sync with the test content.

Another variation on this technique is to have each workgroup maintain their own options file, then submit all of the options files to stride when the aggregated tests are run.

Automating Your Test Organization

The key to reliable and repeatable organization is through automation. If you adopt a standard project-wide naming convention for your test units, a little bit of scripting glue can provide the organization.

At the time the tests are run, the available test unit names, are listed and from these names their parent suites are determined. We create an options file dynamically from this information and submit it to stride.

The naming convention follows the suite pattern: Name segments describing a hierarchy are delimited by a unique character or character sequence.

For C++ environments where namespaces are used, the scope operator ("::") makes a convenient delimiter. For other environments, the underscore ('_') is a good choice.

level1_level2_level3_name

• Have each subgroup manage their own file.

OrganizeIntoSuites.pl

#!/usr/bin/perl
use strict;
use warnings;

# set this to your separator character or string
my $token = '::';

while( <> ) {
    chomp;
    my @segments = split(/$token/);
    # remove the last segment from the array
    my $testunit = pop @segments;
    print "-r /";
    foreach my $segment (@segments) {
        print "$segment/";
    }
    print "($_)\n";
}

Batch File or Shell Script

stride --list --database TestApp.sidb | perl OrganizeIntoSuites.pl > TestsInSuites.txt
stride --database TestApp.sidb --device TCP:localhost:8000 -O TestsInSuites.txt