Test Function List Sample

Introduction

The Test Function List Samples are part of the STRIDE Test Unit Samples. Function Lists are abbreviated as 'FList' in both pragmas (as in scl_test_flist) and documentation. The Test FList Samples pertain to test units that contain lists of functions to be executed. The Test FList functionality is designed to be used for the C language (although it is not restricted from compilation in a C++ environment as well).

The following content relates to the sample files and workspaces installed in %STRIDE_DIR%\Samples\TestUnits\TestFList. This sample consists of a Visual Studio workspace for building a Windows Off-Target App, sample scl_test_flist source code, and a STRIDE workspace for doing more advanced test function list execution.

Getting Started

To begin, open the Visual Studio Solution file in the sample directory. This solution (and corresponding project) were created for Visual Studio 2005. If you have a later version installed, you should be able to open this solution (it will be automatically upgraded if necessary). If you do not have any version of Visual Studio, we recommend you install the current free version of Visual Studio Express.

Once you have successfully opened the solution, rebuild it. The build process has custom STRIDE build rules integrated, The rebuilding will produce a STRIDE database, intercept module source files, and a Windows Off-Target App that incorporates the test class source.

Once the build is complete, perform the following steps to run the test flists in the workspace:

1. launch the Windows Off-Target App, TestFList.exe. This will run in a standard console window.
2. open a command prompt window and change to this sample's directory.
3. at the command prompt, run the command TestUnitRun.pl -v. This will execute all of the test units in the workspace and open a browser to display the results.
4. quit the TestFList.exe application by typing 'q' in its console window.

Sample Test FLists

Now that you have built the Windows Off-Target App and executed the test flists it contains, you can take time to peruse the test flist source and the corresponding results that each produces. This section provides a brief description for each.

Basic Examples

These examples cover the simplest, easiest way to code STRIDE scl_test_flist functionality. These examples use simple POD return types to indicate status and do not annotate the tests with any rich information (such as comments).

01_01_Basic_Simple

This example demonstrates passing and failing tests using the primary POD types that we can infer status from (int, bool, short, and char). The bool type is only accepted in C++ mode.

01_02_Basic_Fixtures

This example shows how to use setup and teardown fixtures. The setup and teardown methods are called immediately before and after the execution of each test method, respectively.

Runtime Services Examples

These examples cover basic usage of our Runtime Test Services API (as declared in srtest.h).

02_01_RuntimeServices_Simple

This example shows how to use srTestCaseSetStatus to set status, srTestCaseAddComment to add a comment, and srTEST_ADD_COMMENT_WITH_LOCATION to add a comment that automatically includes line and file information.

02_02_RuntimeServices_Dynamic

This example shows how to use srTestSuiteAddSuite, srTestSuiteAddTest, srTestSuiteAddAnnotation, and srTestAnnotationAddComment for dynamic suite, test, and annotation creation in the context of a single test method.

02_03_RuntimeServices_Override

This example shows how to use srTestCaseSetStatus to override the status that would otherwise be inferred from the return value.

02_04_RuntimeServices_VarComment

This example demonstrates the use of printf style format strings with srTestCaseAddComment.

Test FList Execution

This sample demonstrates two different techniques for executing scl_test_flist code.

Commandline Execution

Command line execution for TestFlist is done using the TestUnitRun utility. Here are several examples of specific syntax to execute function lists. All of these commands can be invoked from a standard command shell and the arguments shown assume that the commands are executed with the sample's directory as the starting directory. You must have your TestFList.exe application running in order for the runner to be able to initiate a connection to the target simulator. In addition, you should verify that your %STRIDE_DIR%\bin\transport.cfg file is using the TCP transport to connect to port 8000 (these are the default settings when the product is installed).

Simple execution of all test units

The following command executes all of the test units found in the STRIDE database you have previously generated. For the purpose of this sample, since there is only one database, the -d parameter is not strictly needed, but we show it here for completeness.

 TestUnitRun.pl -d TestFList.sidb

This command executes all Test Units found in the (TestFList.sidb) database in descending alpha-numeric sort order. Any Test FList initialization arguments are given default values (typically zero or NULL).

When you run this command, you should see the following:

 Attempting connection using [Sockets (S2)] transport ...
 Connected to device.
 Initializing STRIDE database objects...
 Done.
 Running Test Basic_Fixtures...
 Running Test Basic_Simple...
 Running Test RuntimeServices_Dynamic...
 Running Test RuntimeServices_Override...
 Running Test RuntimeServices_Simple...
 Running Test RuntimeServices_VarComment...
 Disconnected from device.
 Test Results saved to C:\STRIDE\Samples\TestUnits\TestFList\TestFList.xml
 Test Report saved to C:\STRIDE\Samples\TestUnits\TestFList\TestFList.html
 ***************************************************************************
 Results Summary
 ***************************************************************************
   Passed:              17
   Failed:               6
   In Progress:          1
   Not Applicable:       1
   ...in 9 suites.
 ***************************************************************************

Execution based on an order file

TestUnitRun can optionally base its execution on simple text file input. We have provided a simple order file, RunSimple.txt, which specifies a subset of all the function list tests in this sample. This order file also shows how to create subsuites in the final output by using the special {suitepath} syntax, as described in the usage section.

 TestUnitRun.pl -d TestFList.sidb -o RunSimple.txt

...and will produce this output:

 Attempting connection using [Sockets (S2)(debug)] transport ...
 Connected to device.
 Initializing STRIDE database objects...
 Done.
 Running Test Basic_Simple...
 Running Test RuntimeServices_Simple...
 Disconnected from device.
 Test Results saved to C:\s2\seaside\Samples\TestUnits\TestFList\TestFList.xml
 Test Report saved to C:\s2\seaside\Samples\TestUnits\TestFList\TestFList.html
 ***************************************************************************
 Results Summary
 ***************************************************************************
   Passed:               8
   Failed:               4
   In Progress:          1
   Not Applicable:       1
   ...in 4 suites.
 ***************************************************************************

Execution based on file system order

TestUnitRun can also try to infer execution order and suite hierarchy from filesystem organization. If you have organized your function list test source files (only the files that contain the scl_test_flist pragma matter here) in a filesystem hierarchy that you want to mimic in your tests, you can specify a root of a directory tree that contains the relevant test flist source. TestUnitRun will walk the directory structure and determine order and hierarchy of tests based on the directory structure. To see an example of this in action, you can execute this command with the sample:

 TestUnitRun.pl -d TestFList.sidb -f Tests

This will cause the runner to examine the Tests subdirectory structure and build a hierarchy of tests based on that directory tree. Subdirectories will map to suites in the final report. Here is the output for this example:

 Attempting connection using [Sockets (S2)] transport ...
 Connected to device.
 Initializing STRIDE database objects...
 Done.
 Running Test Basic_Simple...
 Running Test Basic_Fixtures...
 Running Test RuntimeServices_Simple...
 Running Test RuntimeServices_Dynamic...
 Running Test RuntimeServices_Override...
 Running Test RuntimeServices_VarComment...
 Disconnected from device.
 Test Results saved to C:\STRIDE\Samples\TestUnits\TestFList\TestFList.xml
 Test Report saved to C:\STRIDE\Samples\TestUnits\TestFList\TestFList.html
 ***************************************************************************
 Results Summary
 ***************************************************************************
   Passed:              17
   Failed:               6
   In Progress:          1
   Not Applicable:       1
   ...in 9 suites.
 ***************************************************************************

Workspace-based Execution

We also provide a sample STRIDE workspace that demonstrates the use of script execution with STRIDE Studio to manage test order and hierarchy. This workspace was created using WorkspaceSetup.pl and the WindowsTestApp framework. The setup and teardown folders provided basic infrastructure scripts that start/stop the simulator application (TestFList.exe) and to manage traceviews used for srPrint message collection. The scripts that drive the testing are in the workspace test folder. What follows is a brief description for each.

RunAll

This folder contains a single script that iterates through the entire collection of test units and executes them one at a time. The order of execution will be in descending alpabetical order (on name) since we explictly called the ArrangeBy collection method.

Run Individual

This folder shows how to use individual scripts to execute scl_test_flist tests. Each script takes the very simple form:

 ascript.TestUnits.Item(TEST_FLIST_NAME).Run();

..where TEST_FLIST_NAME is the name of the scl_test_flist test you want to run. You can then use the Studio tree control to change the order and hierarchy of each item by moving the scripts and creating folders. The sample contains individual scripts for a few of the sample scl_test_flist tests - you are free to move, add, or delete any items as you experiment with the workspace.