Runtime Test Services

From STRIDE Wiki
Revision as of 12:56, 13 June 2011 by Ivailop (talk | contribs) (srTestCaseSetStatus)
Jump to: navigation, search

Runtime Test Services

The Runtime Test Services (declared in srtest.h) are a set of APIs in the STRIDE Runtime that facilitate the writing of target based test code. These APIs make up an optional portion of the STRIDE Runtime and can be used to communicate additional information about tests to the host based reporting mechanism. These APIs also allow target test code to create additional test suites and test cases dynamically at runtime.

There are two ways of accessing these APIs: through C-based functions, or through a C++ base class from which you may derive your C++ test class. These are discussed below in C Test Functions, and C++ Test Classes sections below.

Refer to the SCL Reference Guide or the Runtime Developer's Guide for more detailed information about any of these APIs.

C Test Functions

The following C APIs are provided:


srTestSuiteAddSuite

The srTestSuiteAddSuite()[1] routine is used to add a new test suite to the specified test suite.

srTestSuiteHandle_t srTestSuiteAddSuite(srTestSuiteHandle_t tParent, const srCHAR * szName)
Parameters Type Description
tParent Input Handle to the parent test suite to which new test suite is to be added. srTEST_SUITE_DEFAULT can be used for the default test suite.
szName Input Pointer to a null-terminated string that represents the name of test suite. If null, the default host naming scheme will be used.


Return Value Description
srTestSuiteHandle_t Handle of the newly created test suite. srTEST_SUITE_INVALID indicates failure to create test suite.
#include <srtest.h>

void tfsuite_addSuite(void)
{
  srTestSuiteHandle_t subSuite =
  srTestSuiteAddSuite(srTEST_SUITE_DEFAULT, "tf Sub Suite");
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_addSuite)
#endif

srTestSuiteSetName

The srTestSuiteSetName() routine is used to set the name of the specified test suite.

srWORD srTestSuiteSetName(srTestSuiteHandle_t tTestSuite, const srCHAR * szName)
Parameters Type Description
tTestSuite Input Handle to a test suite. srTEST_SUITE_DEFAULT can be used for the default test suite.
szName Input Pointer to a null-terminated string representing the name of test suite. Cannot be null.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.
#include <srtest.h>

void tfsuite_setName(void)
{
  srTestSuiteSetName(srTEST_SUITE_DEFAULT, "Setting name for default suite");
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_setName)
#endif

srTestSuiteSetDescription

The srTestSuiteSetDescription() routine is used to set the description of the specified test suite.

srWORD srTestSuiteSetDescription(srTestSuiteHandle_t tTestSuite, const srCHAR * szDescr)
Parameters Type Description
tTestSuite Input Handle to a test suite. srTEST_SUITE_DEFAULT can be used for the default test suite.
szDescr Input Pointer to a null-terminated string representing the description of test suite. Cannot be null.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.
#include <srtest.h>
void tfsuite_setDescription(void)
{
  srTestSuiteSetDescription(srTEST_SUITE_DEFAULT,
                            "Setting description for default suite");
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_setDescription)
#endif

srTestSuiteAddCase

The srTestSuiteAddCase()[1] routine is used to add a new test case to the specified test suite.

srTestCaseHandle_t srTestSuiteAddCase(srTestSuiteHandle_t tParent, const srCHAR * szName)
Parameters Type Description
tParent Input Handle to the parent test suite to which new test case is to be added. srTEST_SUITE_DEFAULT can be used for the default test suite.
szName Input Pointer to a null-terminated string that represents the name of test case. If null, the default host naming scheme will be used.


Return Value Description
srTestCaseHandle_t Handle of the newly created test case. srTEST_CASE_INVALID indicates failure to create test case.
#include <srtest.h>
#include <stdio.h>

void tfsuite_addCase(void)
{
  for (int count = 0; count < 5; ++count)
  {
    char szName[25];
    sprintf(szName, "dynamic test case %d", count);
    srTestCaseHandle_t test = srTestSuiteAddCase(srTEST_SUITE_DEFAULT, szName);
    srTEST_ADD_COMMENT_WITH_LOCATION(test, srNULL, "this is a dynamic test case");
    srTestCaseSetStatus(test, srTEST_PASS, 0); 
  }
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_addCase)
#endif

srTestCaseSetName

The srTestCaseSetName() routine is used to set set the name of the specified test case.

srWORD srTestCaseSetName(srTestCaseHandle_t tTestCase, const srCHAR *szName)
Parameters Type Description
tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
szName Input Pointer to a null-terminated string representing the name of test case. Cannot be null.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.
#include <srtest.h>

void tfcase_setName(void)
{
  srTestCaseSetName(srTEST_CASE_DEFAULT, "Setting name for default case");
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_setName)
#endif

srTestCaseSetDescription

The srTestCaseSetDescription() routine is used to set the description of the specified test case.

srWORD srTestCaseSetDescription(srTestCaseHandle_t tTestCase, const srCHAR * szDescr)
Parameters Type Description
tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
szDescr Input Pointer to a null-terminated string representing the description of test case. Cannot be null.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.
#include <srtest.h>

void tfcase_setDescription(void)
{
  srTestCaseSetDescription(srTEST_CASE_DEFAULT,
                           "Setting description for default case");
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_setDescription)
#endif

srTestCaseAddComment

The srTestCaseAddComment() routine is used to add a comment (aka a log) to be reported with the specified test case.

srWORD srTestCaseAddComment(srTestCaseHandle_t tTestCase, const srCHAR * szLabel, const srCHAR * szFmtComment, ...)
Parameters Type Description
tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
szLabel Input Pointer to a null-terminated string specifying the comment's label.
szFmtComment Input Pointer to a null-terminated string, which can be formatted, representing the comment. Cannot be null.
... Input (Optional) Variable argument list to format the comment szFmtComment.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.
#include <srtest.h>

void tfcase_addComment(void)
{
  srTestCaseAddComment(srTEST_CASE_DEFAULT, "Note",
                       "this comment should print %s", "A STRING");
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_addComment)
#endif

srTestCaseSetStatus

The srTestCaseSetStatus() routine is used to set the result of test case.

srWORD srTestCaseSetStatus(srTestCaseHandle_t tTestCase, srTestStatus_e eStatus, srDWORD dwDuration)
Parameters Type Description
tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
eStatus Input Result of the test. Possible values are:
  • srTEST_FAIL
  • srTEST_PASS
  • srTEST_NOTINUSE
  • srTEST_INPROGRESS
  • srTEST_DONE - applicable to dynamic cases - sets the status to pass unless already set to fail or not-in-use
dwDuration Input The duration of the test in clock ticks. Using “0” allows the STRIDE Runtime (Intercept Module) to set the time automatically.


Return Value Description
srOK Success
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
#include <srtest.h>

void tfcase_setStatus(void)
{
  srTestCaseSetStatus(srTEST_CASE_DEFAULT, srTEST_PASS, 0);
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_setStatus)
#endif

srTestCaseSetStatusEx

The srTestCaseSetStatusEx() routine is used to set the result of test case and allow specification of an extendedFailureCode.

srWORD srTestCaseSetStatus(srTestCaseHandle_t tTestCase, srTestStatus_e eStatus, srDWORD dwDuration, srLONG lExtendedFailureCode)
Parameters Type Description
tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
eStatus Input Result of the test.
dwDuration Input The duration of the test in clock ticks. Using “0” allows the STRIDE Runtime (Intercept Module) to set the time automatically.
lExtendedFailureCode Input The Stride framework uses the extendedFailureCode to capture the numeric results of test method when the test method fails via a numeric (non-void, nonbool) return type.


Return Value Description
srOK Success
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
#include <srtest.h>

void tfcase_setStatusEx(void)
{
  srTestCaseSetStatusEx(srTEST_CASE_DEFAULT, srTEST_FAIL, 0, -5);
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_setStatusEx)
#endif

srTestSuiteAddAnnotation

The srTestSuiteAddAnnotation() routine is used to add a new annotation to the specified test suite.

srTestAnnotationHandle_t srTestSuiteAddAnnotation(rTestSuiteHandle_t tParent, srTestAnnotationLevel_e eLevel, const srCHAR * szName, const srCHAR * szDescr)
Parameters Type Description
tParent Input Handle to the parent test suite to which new annotation is to be added. srTEST_SUITE_DEFAULT can be used for the default test suite.
eLevel Input Annotation level. Cannot be empty.
szName Input Pointer to a null-terminated string that represents the name of annotation. If null, the default host naming scheme will be used.
szDescr Input Pointer to a null-terminated string representing the description of annotation. If null, description will be empty.


Return Value Description
srTestAnnotationHandle_t HHandle of the newly created annotation. srTEST_ANNOTATION_INVALID indicates failure to create annotation.
#include <srtest.h>

void tfsuite_addAnnotation(void) 
{
  for (int count = 0; count < 5; ++count)
  {
    char szName[25];
    sprintf(szName, "annotation %d", count);
    srTestAnnotationHandle_t annot = 
                     srTestSuiteAddAnnotation(srTEST_SUITE_DEFAULT,
                                              srTEST_ANNOTATION_LEVEL_ERROR,
                                              szName,
                                              "description of annotation");
  }
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_addAnnotation)
#endif

srTestCaseAddAnnotation

The srTestCaseAddAnnotation() routine is used to add a new annotation to the specified test case.

srTestAnnotationHandle_t srTestCaseAddAnnotation(rTestCaseHandle_t tParent, srTestAnnotationLevel_e eLevel, const srCHAR * szName, const srCHAR * szDescr)
Parameters Type Description
tParent Input Handle to the parent test case to which new annotation is to be added. srTEST_CASE_DEFAULT can be used for the default test case.
eLevel Input Annotation level. Cannot be empty.
szName Input Pointer to a null-terminated string that represents the name of annotation. If null, the default host naming scheme will be used.
szDescr Input Pointer to a null-terminated string representing the description of annotation. If null, description will be empty.


Return Value Description
srTestAnnotationHandle_t HHandle of the newly created annotation. srTEST_ANNOTATION_INVALID indicates failure to create annotation.
#include <srtest.h>

void tfcase_addAnnotation(void) 
{
  for (int count = 0; count < 5; ++count)
  {
    char szName[25];
    sprintf(szName, "annotation %d", count);
    srTestAnnotationHandle_t annot = 
                     srTestCaseAddAnnotation(srTEST_CASE_DEFAULT,
                                             srTEST_ANNOTATION_LEVEL_ERROR,
                                             szName,
                                             "description of annotation");
  }
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_addAnnotation)
#endif

srTestAnnotationAddComment

The srTestAnnotationAddComment() routine is used to add a comment (aka a log) to be reported with the specified annotation.

srWORD srTestAnnotationAddComment(srTestAnnotationHandle_t tTestAnnotation, const srCHAR * szLabel, const srCHAR * szFmtComment, ...)
Parameters Type Description
tTestAnnotation Input Handle to an annotation created using srTestSuiteAddAnnotation.
szLabel Input Pointer to a null-terminated string for the label. If null, the default label will be applied.
szFmtComment Input Pointer to a null-terminated string, which can be formatted, representing the comment. Cannot be null.
... Input (Optional) Variable argument list to format the comment szFmtComment.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.
#include <srtest.h>

void tfsuiteAnnotation_addComment(void) 
{
  srTestAnnotationHandle_t annot = 
                           srTestSuiteAddAnnotation(srTEST_SUITE_DEFAULT,
                                                    srTEST_ANNOTATION_LEVEL_ERROR,
                                                    annot,
                                                    annot description);
  srTestAnnotationAddComment(annot,
                             srNULL,
                             "this comment should print %s", "A STRING");
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuiteAnnotation_addComment)
#endif

C++ Test Classes

The Runtime Test Services C APIs work equally well from C test functions and C++ test classes. If, however, you are using C++ it might be more convinient for you to derive your C++ test classes from the Runtime Test Services C++ base class, srTest. That way you will have access to a set of simpler to use C++ classes.

The following C++ classes are provided:

class srTest

The srTest class provides two Member Objects:

  • testSuite of class srTestSuite
  • testCase of class srTestCase

class srTestSuite

The srTest class provides the folowing methods:

method AddSuite

The AddSuite[1] method is used to add a new test suite.

srTestSuite AddSuite(const srCHAR * szName = srNULL)
Parameters Type Description
szName Input (Optional) Pointer to null-terminated string that represents the name of test suite. If empty, the default host naming scheme will be used.


Return Value Description
srTestSuite Newly created test suite class.
#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void suiteAddSuite()
  {
    stride::srTestSuite suite = testSuite.AddSuite("tc Sub Suite");
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif
method SetName

The SetName method is used to set the name of test suite.

srWORD SetName(const srCHAR * szName)
Parameters Type Description
szName Input Pointer to null-terminated string representing the name of test suite. Cannot be empty.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.
#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void suiteSetName()
  {
    testSuite.SetName("Setting name for suite");
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif
method SetDescription

The SetDescription method is used to set the description of test suite.

srWORD SetDescription(const srCHAR * szDescr)
Parameters Type Description
szDescr Input Pointer to null-terminated string representing the description of test suite. Cannot be empty.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.
#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void suiteSetDescription()
  {
    testSuite.SetDescription("Setting description for suite");
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif
method AddCase

The AddCase[1] method is used to add a new test case to test suite.

srTestCase AddCase(const srCHAR * szName = srNULL)
Parameters Type Description
szName Input (Optional) Pointer to null-terminated string that represents the name of test case. If empty, the default host naming scheme will be used.


Return Value Description
srTestCase Newly created test case class.


Example

#include <srtest.h>
#include <sstream>

class srtest_class : public stride::srTest
{
public:
  void suiteAddSuite()
  {
    const std::string prefix("dynamic test case ");
    for (int count = 0; count < 5; ++count)
    {
      std::stringstream strm;
      strm << prefix << count;
      stride::srTestCase tc = testSuite.AddCase(strm.str().c_str());
      tc.AddComment("this is a dynamic test case");
      tc.SetStatus(srTEST_PASS);
    }
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif
method AddAnnotation

The AddAnnotation method is used to add a new annotation to test suite.

srTestAnnotation AddAnnotation(srTestAnnotationLevel_e eLevel, const srCHAR * szName = srNULL, const srCHAR * szDescr = srNULL)
Parameters Type Description
eLevel Input Annotation level. Cannot be empty.
szName Input (Optional) Pointer to null-terminated string that represents the name of annotation. If empty, the default host naming scheme will be used.
szDescr Input (Optional) Pointer to null-terminated string that represents the description of annotation. If empty, the description will be blank.


Return Value Description
srTestAnnotation Newly created annotation class.
#include <srtest.h>
#include <sstream>

class srtest_class : public stride::srTest
{
public:
  void suiteAddAnnotation()
  {
    for (int count = 0; count < 5; ++count)
    {
      std::stringstream strmName;
      std::stringstream strmDescr;
      strmName << "annotation " << count;
      strmDescr << "description of annotation " << count;
      stride::srTestAnnotation ta = 
                 testSuite.AddAnnotation(srTEST_ANNOTATION_LEVEL_INFO,
                                         strmName.str().c_str(),
                                         strmDescr.str().c_str());
    }
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif

class srTestCase

The srTestCase class provides the following methods:

method SetName

The SetName method is used to set the name of test case.

srWORD SetName(const srCHAR * szName)
Parameters Type Description
szName Input Pointer to null-terminated string representing the name of test case. Cannot be empty.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.
#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void caseSetName()
  {
    testCase.SetName("Setting name for case");
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif
method SetDescription

The SetDescription method is used to set the description of test case.

srWORD SetDescription(const srCHAR * szDescr)
Parameters Type Description
szDescr Input Pointer to null-terminated string representing the description of test case. Cannot be empty.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.
#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void caseSetDescription()
  {
    testCase.SetDescription("Setting description for case");
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif
method AddComment

The AddComment method is used to add a comment to test case.

srWORD AddComment(const srCHAR * szFmtComment, ...)
Parameters Type Description
szFmtComment Input Pointer to null-terminated string, which can be formatted, representing the comment. Cannot be empty.
... Input (Optional) Variable argument list to format the comment szFmtComment.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.
#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void caseAddComment()
  {
    testCase.AddComment("this comment should print %s", "A STRING");
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif
method SetStatus

The SetStatus method is used to set the result of the default test case.

srWORD SetStatus(srTestStatus_e eStatus, srDWORD dwDuration = 0)
Parameters Type Description
eStatus Input Result of the test.
dwDuration Input (Optional) The duration of the test in clock ticks. The default is 0.


Return Value Description
srOK Success
srTEST_ERR_HANDLE_INVALID Invalid handle.
#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void caseSetStatus()
  {
    testCase.SetStatus(srTEST_INPROGRESS, 0);
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif
method AddAnnotation

The AddAnnotation method is used to add a new annotation to test case.

srTestAnnotation AddAnnotation(srTestAnnotationLevel_e eLevel, const srCHAR * szName = srNULL, const srCHAR * szDescr = srNULL)
Parameters Type Description
eLevel Input Annotation level. Cannot be empty.
szName Input (Optional) Pointer to null-terminated string that represents the name of annotation. If empty, the default host naming scheme will be used.
szDescr Input (Optional) Pointer to null-terminated string that represents the description of annotation. If empty, the description will be blank.


Return Value Description
srTestAnnotation Newly created annotation class.
#include <srtest.h>
#include <sstream>

class srtest_class : public stride::srTest
{
public:
  void caseAddAnnotation()
  {
    for (int count = 0; count < 5; ++count)
    {
      std::stringstream strmName;
      std::stringstream strmDescr;
      strmName << "annotation " << count;
      strmDescr << "description of annotation " << count;
      stride::srTestAnnotation ta = 
                 testCase.AddAnnotation(srTEST_ANNOTATION_LEVEL_INFO,
                                         strmName.str().c_str(),
                                         strmDescr.str().c_str());
    }
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif

class srTestAnnotation

The srTestAnnotation class provides the following methods:

method AddComment

The AddComment method is used to add a comment to an annotation created under a test suite.

srWORD AddComment(const srCHAR * szLabel, const srCHAR * szFmtComment, ...)
Parameters Type Description
szLabel Input Pointer to null-terminated string for the label. If null, the default label will be applied.
szFmtComment Input Pointer to null-terminated string, which can be formatted, representing the comment. Cannot be empty.
... Input (Optional) Variable argument list to format the comment szFmtComment.


Return Value Description
srOK Success
srERR Null string passed in.
srTEST_ERR_HANDLE_INVALID Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED String passed in was too large and was truncated.
#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void suiteAnnotationAddComment()
  {
    stride::srTestAnnotation ta = 
                 testSuite.AddAnnotation(srTEST_ANNOTATION_LEVEL_INFO,
                                         "annot",
                                         "annot description");
    ta.AddComment("this comment on annotation should print %s", "A STRING");
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif

Notes

  1. 1.0 1.1 1.2 1.3 1.4 1.5 The use of these functions in the context of any test method will cause the STRIDE Test Unit framework to delete the default test case that was created for the current test method since their use implies that dynamic test cases will be created. In addition, the use of the dynamic test case creation methods will cause the framework to interpret the srTEST_CASE_DEFAULT value as the most recently created test case.