Test Units Overview

From STRIDE Wiki
Revision as of 15:15, 7 July 2008 by Stevel (talk | contribs) (→‎srTestCaseAddComment)
Jump to: navigation, search

Introduction

STRIDE enables testing of C++ code through the use of xUnit-style test classes. Test classes can be written by developers, captured using an SCL pragma, and executed from the host. STRIDE facilitates the execution of some or all of the test classes by automatically creating entry points for the execution of test classes on the target.

Using C++ test classes

Prerequisites

see Perl requirements.

How to get started (Overview)

The required steps to get started with writing C++ test classes are as follows:

  1. Create a new Studio workspace (or open an existing one).
  2. Set the workspace to compile in C++ mode (In Studio, choose Tools->Settings->Compile as Cpp).
  3. Write a test class. Click here for more information on creating test classes. #include <srtest.h> class Simple { public: int test1(void) { return 0;} // PASS int test2(void) { return 23;} // FAIL <>0 }; #pragma scl_test_class(Simple)
  4. Compile the workspace & review the Simple interface in the Studio Interface tab
  5. Create a script to generate the Intercept Module(IM) after the compilation step.
    For the simple STUB generation required for C++ test class execution, you can use the following code (perl syntax)
  use strict; 
  use Win32::OLE qw(in); 
  Win32::OLE->Option(Warn => 3); 
  my $intercept = $main::studio->Workspace->Intercept; $intercept->{Path} = $main::studio->Workspace->Path; $intercept->{Name} = $main::studio->Workspace->Name; map {$intercept->Item($_)->{Stub} = 1} (0..($intercept->Count - 1)); $intercept->Create();
  1. Optionally add custom scripts to automate the building and executing your application. Refer to Using Frameworks for more information regarding building and executing tests with a target device.
  2. Ensure that the Studio workspace include path contains the location to all of your test class declaration (header) files.
  3. Once you have created one or more test classes, ensure the following:
    • Workspace is compiled and saved
    • Intercept Module is generated (Stubs for all Test Classes)
    • Target application re-built
    • Target application downloaded & started
    • STRIDE Connected to Target
  4. If your application is running, you can start executing test classes.
    • You can test-execute individual test classes interactively using the Studio interface view. To do this, open the user interface view corresponding to the test class you would like to execute, then call it. The return values will indicate how many tests produced each of four (4) result types. Furthermore, the input to the entry point will allow you to select all methods for execution (the default) or individual methods via a dropdown list of enumerated values.
    • Once you are confident that the test classes are behaving as expected, you can generate one or more execution scripts using the Script Wizard. Sample templates for executing test class entry points are provided in the %STRIDE_DIR%\templates\Script Wizard directory.
    • There are several installed library components that provide some of the core functionality for test class harnessing and execution. These components are installed and registered with the core product installation. For sample examples, see the Test Class Component that can be used for running test classes from scripts.
  5. For integration with larger regression test workspaces, we recommend that developers check in their test class code and, optionally, the template-generated scripts that can be used to execute their test classes.

Pragmas for Test Classes

STRIDE supports three pragmas for capturing and qualifying test classes:

  • scl_test_class ( class ): Declares a test class as captured. Once captured, STRIDE will generate the appropriate code for executing the test methods in the class.
  • scl_test_setup ( class , method ): [optional] Declares a member method to be a setup fixture for the class. If specified, the setup method will be called before the execution of each test method.
  • scl_test_teardown ( class , method ): [optional] Declares a member method to be a teardown fixture for the class. If specified, the teardown method will be called after the execution of each test method.

C++ test class requirements

Several variations on typical xUnit-style test classes are supported. The additional supported features include:

  • Test status can be set using STRIDE Runtime APIs or by specifying simple return types for test methods.
  • Simple return types: 0 = PASS; <> 0 = FAIL
  • void return type with no explict status setting is assumed PASS
  • Test writers can create additional child suites and tests at runtime by using Runtime APIs.
  • We do not rely on exceptions for reporting of status.

The STRIDE test class framework has the following requirements of each test class:

  • The test class must have a suitable default (no-argument) constructor.
  • The test class must have one or more public methods suitable as test methods. Allowable test methods always take no arguments (void) and return either void or simple integer types (int, short, long, char or bool). At this time, we do not allow typedef types or macros for the return values specification.
  • the scl_test_class pragma must be applied to the class.

Simple example using return values for status

 #include <srtest.h>
 
 class Simple {
 public:
   int tc_Int_ExpectPass(void) {return 0;}
   int tc_Int_ExpectFail(void) {return -1;}
   bool tc_Bool_ExpectPass(void) {return true;}
   bool tc_Bool_ExpectFail(void) {return false;}
 };
 #ifdef _SCL
 #pragma scl_test_class(Simple)
 #endif

Simple example using runtime test service APIs

 #include <srtest.h>
 
 class RuntimeServices_basic {
 public: 
   void tc_ExpectPass(void) 
   {
       srTestCaseAddComment(srTEST_CASE_DEFAULT, "this test should pass");
       srTestCaseSetStatus(srTEST_CASE_DEFAULT, srTEST_PASS, 0); 
   }
   void tc_ExpectFail(void) 
   {
       srTestCaseAddComment(srTEST_CASE_DEFAULT, "this test should fail");
       srTestCaseSetStatus(srTEST_CASE_DEFAULT, srTEST_FAIL, 0); 
   }
   void tc_ExpectInProgress(void) 
   {
       srTestCaseAddComment(srTEST_CASE_DEFAULT, "this test should be in progress");
   }
 };
 #ifdef _SCL
 #pragma scl_test_class(RuntimeServices_basic)
 #endif

Simple example using srTest base class

 #include <srtest.h>
 
 class MyTest : public stride::srTest {
 public:
   void tc_ExpectPass(void) 
   {
       testCase.AddComment("this test should pass");
       testCase.SetStatus(srTEST_PASS, 0); 
   }
   void tc_ExpectFail(void) 
   {
       testCase.AddComment("this test should fail");
       testCase.SetStatus(srTEST_FAIL, 0); 
   }
   void tc_ExpectInProgress(void) 
   {
       testCase.AddComment("this test should be in progress");
   }
   int tc_ChangeMyName(void) 
   {
       testCase.AddComment("this test should have name = MyChangedName");
       testCase.SetName("MyChangedName");
       return 0;
   }
   int tc_ChangeMyDescription(void) 
   {
       testCase.AddComment("this test should have a description set");
       testCase.SetDescription("this is my new description");
       return 0;
   }
 };
 #ifdef _SCL
 #pragma scl_test_class(MyTest)
 #endif

Runtime Test Services

The Runtime Test Services (declared in srTest.h) are a set of APIs in the STRIDE Runtime that facilitate the writing of target based test code. These APIs make up an optional portion of the STRIDE Runtime and can be used to communicate additional information about tests to the host based reporting mechanism. These APIs also allow target test code to create additional test suites and test cases dynamically at runtime.

The following C APIs are provided:

srTestSuiteAddSuite

Prototype 

 srTestSuiteHandle_t srTestSuiteAddSuite(srTestSuiteHandle_t tParent, const srCHAR * szName)

Description The srTestSuiteAddSuite() routine is used to add a new test suite to the specified test suite.

Parameters Type Description
tParent Input Handle to the parent test suite to which new test suite is to be added. srTEST_SUITE_DEFAULT can be used for the default test suite.
szName Input Pointer to a null-terminated string that represents the name of test suite. If null, the default host naming scheme will be used.


Return Value Description
srTestSuiteHandle_t Handle of the newly created test suite. srTEST_SUITE_INVALID indicates failure to create test suite.


Example 

 #include "srtest.h"
 void tfsuite_addSuite(void)
 {
   srTestSuiteHandle_t subSuite =
   srTestSuiteAddSuite(srTEST_SUITE_DEFAULT, "tf Sub Suite");
 }

 #ifdef _SCL
 #pragma scl_test_flist(“testfunc”, tfsuite_addSuite)
 #endif


srTestSuiteSetName

Prototype 

 srWORD srTestSuiteSetName(srTestSuiteHandle_t tTestSuite, const srCHAR * szName)

Description The srTestSuiteSetName() routine is used to set the name of the specified test suite.

Parameters Type Description
tTestSuite Input Handle to a test suite. srTEST_SUITE_DEFAULT can be used for the default test suite.
szName Input Pointer to a null-terminated string representing the name of test suite. Cannot be null.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.


Example 

 #include "srtest.h"
 void tfsuite_setName(void)
 {
   srTestSuiteSetName(srTEST_SUITE_DEFAULT, "Setting name for default suite");
 } 

 #ifdef _SCL
 #pragma scl_test_flist(“testfunc”, tfsuite_setName)
 #endif

srTestSuiteSetDescription

Prototype 

 srWORD srTestSuiteSetDescription(srTestSuiteHandle_t tTestSuite, const srCHAR * szDescr)

Description The srTestSuiteSetDescription() routine is used to set the description of the specified test suite.


Parameters Type Description
tTestSuite Input Handle to a test suite. srTEST_SUITE_DEFAULT can be used for the default test suite.
szDescr Input Pointer to a null-terminated string representing the description of test suite. Cannot be null.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.


Example 

 #include "srtest.h"
 void tfsuite_setDescription(void)
 {
   srTestSuiteSetDescription(srTEST_SUITE_DEFAULT,
                             "Setting description for default suite");
 } 

 #ifdef _SCL
 #pragma scl_test_flist(“testfunc”, tfsuite_setDescription)
 #endif

srTestSuiteAddTest

Prototype 

 srTestCaseHandle_t srTestSuiteAddTest(srTestSuiteHandle_t tParent, const srCHAR * szName)

Description The srTestSuiteAddTest() routine is used to add a new test case to the specified test suite.


Parameters Type Description
tParent Input Handle to the parent test suite to which new test case is to be added. srTEST_SUITE_DEFAULT can be used for the default test suite.
szName Input Pointer to a null-terminated string that represents the name of test case. If null, the default host naming scheme will be used.


Return Value Description
srTestCaseHandle_t Handle of the newly created test case. srTEST_CASE_INVALID indicates failure to create test case.


Example 

 #include "srtest.h"
 #include <sstream>
 void tfsuite_addTest(void)
 {
   const std::string prefix("dynamic test case ");
   for (int count = 0; count < 5; ++count)
   {
     std::stringstream strm;
     strm << prefix << count;
     srTestCaseHandle_t test = srTestSuiteAddTest(srTEST_SUITE_DEFAULT,
                                                  strm.str().c_str());
     srTEST_ADD_COMMENT_WITH_LOCATION(test, "this is a dynamic test case");
   }
 }

 #ifdef _SCL
 #pragma scl_test_flist(“testfunc”, tfsuite_addTest)
 #endif

srTestCaseSetName

Prototype 

 srWORD srTestCaseSetName(srTestCaseHandle_t tTestCase, const srCHAR *szName)

Description The srTestCaseSetName() routine is used to set set the name of the specified test case.


Parameters Type Description
tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
szName Input Pointer to a null-terminated string representing the name of test case. Cannot be null.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.


Example 

 #include "srtest.h"
 #include <sstream>
 void tfcase_setName(void)
 {
   srTestCaseSetName(srTEST_CASE_DEFAULT, "Setting name for default case");
 } 

 #ifdef _SCL
 #pragma scl_test_flist(“testfunc”, tfcase_setName)
 #endif

srTestCaseSetDescription

Prototype 

 srWORD srTestCaseSetDescription(srTestCaseHandle_t tTestCase, const srCHAR * szDescr)

Description The srTestCaseSetDescription() routine is used to set the description of the specified test case.


Parameters Type Description
tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
szDescr Input Pointer to a null-terminated string representing the description of test case. Cannot be null.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.


Example 

 #include "srtest.h"
 void tfcase_setDescription(void)
 {
   srTestCaseSetDescription(srTEST_CASE_DEFAULT,
                            "Setting description for default case");
 } 

 #ifdef _SCL
 #pragma scl_test_flist(“testfunc”, tfcase_setDescription)
 #endif

srTestCaseAddComment

Prototype 

 srWORD srTestCaseAddComment(srTestCaseHandle_t tTestCase, const srCHAR * szFmtComment, ...)

Description The srTestCaseAddComment() routine is used to add a comment (aka a log) to be reported with the specified test case.


Parameters Type Description
tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
szFmtComment Input Pointer to a null-terminated string, which can be formatted, representing the comment. Cannot be null.
... Input (Optional) Variable argument list to format the comment szFmtComment.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.


Example 

 #include "srtest.h"
 void tfcase_addComment(void)
 {
   srTestCaseAddComment(srTEST_CASE_DEFAULT,
                        "this comment should print %s", "A STRING");
 }

 #ifdef _SCL
 #pragma scl_test_flist(“testfunc”, tfcase_addComment)
 #endif

srTestCaseSetStatus

Prototype 

 srWORD srTestCaseSetStatus(srTestCaseHandle_t tTestCase, srTestStatus_e eStatus, srDWORD dwDuration)

Description The srTestCaseSetStatus() routine is used to set the result of test case.


Parameters Type Description
tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
eStatus Input Result of the test.
dwDuration Input The duration of the test in clock ticks. Using “0” allows the STRIDE Runtime (Intercept Module) to set the time automatically.


Return Value Description
srOK Success
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.


Example 

 #include "srtest.h"
 void tfcase_setStatus(void)
 {
   srTestCaseSetStatus(srTEST_CASE_DEFAULT, srTEST_PASS, 0);
 }

 #ifdef _SCL
 #pragma scl_test_flist(“testfunc”, tfcase_setStatus)
 #endif

srTestCaseSetStatusEx

Prototype 

 srWORD srTestCaseSetStatus(srTestCaseHandle_t tTestCase, srTestStatus_e eStatus, srDWORD dwDuration, srLONG lExtendedFailureCode)

Description The srTestCaseSetStatusEx() routine is used to set the result of test case and allow specification of an extendedFailureCode.


Parameters Type Description
tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
eStatus Input Result of the test. dwDuration Input The duration of the test in clock ticks. Using “0” allows the STRIDE Runtime (Intercept Module) to set the time automatically.
lExtendedFailureCode Input The Stride framework uses the extendedFailureCode to capture the numeric results of test method when the test method fails via a numeric (non-void, nonbool) return type.


Return Value Description
srOK Success
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.


Example 

 #include "srtest.h"
 void tfcase_setStatusEx(void)
 {
   srTestCaseSetStatusEx(srTEST_CASE_DEFAULT, srTEST_FAIL, 0, -5);
 }

 #ifdef _SCL
 #pragma scl_test_flist(“testfunc”, tfcase_setStatusEx)
 #endif
  • srTestSuiteAddSuite: creates an additional sub-suite at runtime.
  • srTestSuiteSetName: sets the name of the specified suite.
  • srTestSuiteSetDescription: sets the description of the specified suite.
  • srTestSuiteAddTest: creates an additional test case at runtime.
  • srTestCaseSetName: sets the name of the specified test case.
  • srTestCaseSetDescription: sets the description of the specified test case.
  • srTestCaseAddComment: adds a comment to the specified test case.
  • srTestCaseSetStatus: explicitly sets the status for the specified test case.

These C APIs work equally well from C test functions and C++ test classes. If, however, you choose to derive your C++ test classes from the STRIDE Runtime base class, srTest, then you will have access to member objects in srTest and their methods that provide the same functionality as the C API. The srTest base class provides two Member Objects, via which you can access functionality:

Member Objects:

  • testSuite, which has methods:
    • AddSuite
    • SetName
    • SetDescription
    • AddTest
  • testCase, which has methods:
    • SetName
    • SetDescription
    • AddComment
    • SetStatus

Refer to the Reference Guide or the Runtime Developers Guide, both available in the STRIDE Online Help, for detailed information about any of these functions.

Scripting a Test Class

To automate the execution and reporting of a Test Class a script is required. Scripts can be written by hand or automatically generated using the Script Wizard and a corresponding template script. An additional requirement for test class scripting is the usage of the STRIDE.testclass component. The test class component assembles all of the reporting information for the test class and its corresponding test methods.

  • Require useage of STRIDE.testclass
  • Can be written by hand (refer below)
  • Can leverage Templates via the Script Wizard
  • Order of multiple test classes dictated by SUID assignment


JScript example for a single test class

The following example script is used to harness a test class that has been captured using #pragma scl_test_class(Simple). 

  // Ensure test class exists
  if (ascript.TestUnits.Item("Simple") != null ) 
    ascript.TestUnits.Item("Simple").Run();

Perl example for a single test class

The following example script is used to harness a test class that has been captured using #pragma scl_test_class(Simple). 


  use strict;
  use Win32::OLE;
  Win32::OLE->Option(Warn => 3);
  
  my $tu = $main::ascript->TestUnits->Item("Simple");
  if (defined $tu) {
    $tu->Run();
  }

JScript example for multiple test classes

The following example script is used to harness two test classes that has been captured using #pragma scl_test_class(Simple1) and #pragma scl_test_class(Simple2). 


  var Functions = ["Simple1","Simple2"];
 
  // iterate through each function
  for (i in Functions)
  {
    var tu = ascript.TestUnits.Item(Functions[i]);
    if ( tu != null ) 
      tu.Run();
  }

Perl example for multiple test classes

The following example script is used to harness two test classes that has been captured using #pragma scl_test_class(Simple1) and #pragma scl_test_class(Simple2). 


  use strict;
  use Win32::OLE;
  Win32::OLE->Option(Warn => 3);
  
  # initialize an array with all selected function names
  my @FunctionNames = ("Simple1","Simple2");
  foreach (@FunctionNames) {   
    my $tu = $main::ascript->TestUnits->Item($_->[1]);
    die "TestUnit not found: $_->[1]\n" unless (defined $tu);
    $tu->Run();
  }
Retrieved from "https://www.stridewiki.com/index.php?title=Test_Units_Overview&oldid=5838"