Runtime Test Services

From STRIDE Wiki
Jump to: navigation, search

The Runtime Test Services (declared in srtest.h) are a set of APIs in the Stride Runtime that facilitate the writing of target based test code. These APIs make up an optional portion of the Stride Runtime and can be used to communicate additional information about tests to the host based reporting mechanism. These APIs also allow target test code to create additional test suites and test cases dynamically at runtime.

There are two ways of accessing these APIs: through C-based functions, or through a C++ base class from which you may derive your C++ test class. These are discussed below in C Test Functions, and C++ Test Classes sections below.

C Test Functions

The following C APIs are provided:


srTestGetParam

The srTestGetParam() function is used to get the value of a named parameter as a string.

srWORD srTestGetParam(const srCHAR * szName, srCHAR * szValue, srDWORD dwSize, const srCHAR * szDefValue);


Parameters Type Description
szName Input Pointer to a null-terminated string that represents the name of the parameter. Cannot be null.
szValue Output Pointer to a block of memory to store the value of the parameter with a maximum size of dwSize chars
dwSize Input Size in chars of the buffer pointed to by szValue
szDefValue Input (optional) Pointer to a null-terminated string that represents a default value in case the parameter is not specified. By setting this value to srNULL (or omitting it), you can use srTestGetParam() to test for the existence of a named parameter. If the named parameter doesn't exist, the function will return srERR_HANDLE_INVALID.


Return Value Description
srOK Success
srERR Null or empty szName string passed in.
srERR_HANDLE_INVALID The named parameter is not found and szDefValue is set to srNULL.


#include <srtest.h>
 
#define MAX_VALUE_LEN 128
void tfsuite_getParam(void)
{
  srCHAR szValue[MAX_VALUE_LEN] = {0};
 
  srWORD wRet = srTestGetParam("ParamName", szValue, MAX_VALUE_LEN, "default value");
}
 
#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_getParam)
#endif


srTestGetParamLong

The srTestGetParamLong() function is used to get the value of a named parameter as a long.

srLONG srTestGetParamLong(const srCHAR * szName, srLONG lDefValue);


Parameters Type Description
szName Input Pointer to a null-terminated string that represents the name of the parameter.
lDefValue Input Default value that is returned if the parameter is not found.


Return Value Description
srLONG Parameter value, or value specified by lDefValue if parameter isn't found.


#include <srtest.h>
 
void tfsuite_getParam(void)
{
  srLONG lVal = srTestGetParamLong("ParamName", -1);
}
 
#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_getParam)
#endif


srTestGetParamDouble

The srTestGetParamDouble() function is used to get the value of a named parameter as a double.

double srTestGetParamDouble(const srCHAR * szName, double dbDefValue);


Parameters Type Description
szName Input Pointer to a null-terminated string that represents the name of the parameter.
dbDefValue Input Default value that is returned if the parameter is not found.


Return Value Description
double Parameter value, or value specified by dbDefValue if parameter isn't found.


#include <srtest.h>
 
void tfsuite_getParam(void)
{
  double dbVal = srTestGetParamDouble("ParamName", 0.5772156649);
}
 
#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_getParam)
#endif


srTestSuiteAddCase

The srTestSuiteAddCase() routine is used to add a new test case to the specified test suite.

srTestCaseHandle_t srTestSuiteAddCase(const srCHAR * szName, const srCHAR * szDescr)


Parameters Type Description
szName Input Pointer to a null-terminated string that represents the name of the new test case. If null, the default host naming scheme will be used.
szDescr Input Pointer to a null-terminated string representing the description of the new test case. If null, description will be empty.


Return Value Description
srTestCaseHandle_t Handle of the newly created test case. srTEST_CASE_INVALID indicates failure to create a test case.


#include <srtest.h>
#include <stdio.h>
 
void tfsuite_addCase(void)
{
  for (int count = 0; count < 5; ++count)
  {
    char szName[25];
    sprintf(szName, "dynamic test case %d", count);
    srTestCaseHandle_t test = srTestSuiteAddCase(szName, "this is a dynamic test case");
    srTestCaseSetStatus(test, srTEST_PASS, 0); 
  }
}
 
#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_addCase)
#endif


srTestSuiteAddAnnotation

The srTestSuiteAddAnnotation() routine is used to add a new annotation to the specified test suite.

srTestAnnotationHandle_t srTestSuiteAddAnnotation(srTestAnnotationLevel_e eLevel,
                                                  const srCHAR * szName, 
                                                  const srCHAR * szFmtDescr, ...)


Parameters Type Description
eLevel Input Annotation level.
  • srTEST_ANNOTATION_LEVEL_TRACE,
  • srTEST_ANNOTATION_LEVEL_DEBUG,
  • srTEST_ANNOTATION_LEVEL_INFO,
  • srTEST_ANNOTATION_LEVEL_WARN,
  • srTEST_ANNOTATION_LEVEL_ERROR,
  • srTEST_ANNOTATION_LEVEL_FATAL
szName Input Pointer to a null-terminated string representing the name of the new annotation. If null, the default host naming scheme will be used.
szFmtDescr Input Pointer to a null-terminated string representing the description of the new annotation. Cannot be null.
... Input (Optional) Variable argument list to format szFmtDescr.


Return Value Description
srTestAnnotationHandle_t Handle of the newly created test annotation. srTEST_ANNOTATION_INVALID indicates failure to create a test annotation.


#include <srtest.h>
 
void tfsuite_addAnnotation(void) 
{
  for (int count = 0; count < 5; ++count)
  {
    char szName[25];
    sprintf(szName, "annotation %d", count);
    srTestAnnotationHandle_t annot = 
                     srTestSuiteAddAnnotation(srTEST_ANNOTATION_LEVEL_ERROR,
                                              szName,
                                              "description of annotation");
  }
}
 
#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_addAnnotation)
#endif


srTestSuiteSetData

The srTestSuiteSetData() routine is used to associate a custom name|value pair with a test suite.

srWORD srTestSuiteSetData(const srCHAR * szName, const srCHAR * szFmtValue, ...)


Parameters Type Description
szName Input Pointer to a null-terminated string representing the name of the custom pair. Cannot be null.
szFmtValue Input Pointer to a null-terminated string representing the value of the custom pair. Cannot be null.
... Input (Optional) Variable argument list to format szFmtValue.


Return Value Description
srWORD srOK on success, srERR_HANDLE_INVALID on invalid handle.


#include <srtest.h>
 
void tfcase_setData(void)
{
  srTestSuiteSetData("MyName", "my value");
}
 
#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_setData)
#endif


srTestCaseSetStatus

The srTestCaseSetStatus() routine is used to set the result of test case.

srWORD srTestCaseSetStatus(srTestCaseHandle_t tTestCase, srTestStatus_e eStatus, srDWORD dwDuration)


Parameters Type Description
tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
eStatus Input Result of the test. Possible values are:
  • srTEST_FAIL
  • srTEST_PASS
  • srTEST_NOTINUSE
  • srTEST_INPROGRESS
  • srTEST_DONE - applicable to dynamic cases - sets the status to pass unless already set to fail or not-in-use
dwDuration Input The duration of the test in clock ticks. Using "0" allows the Stride Runtime (Intercept Module) to set the time automatically.


Return Value Description
srWORD srOK on success, srERR_HANDLE_INVALID on invalid handle.


#include <srtest.h>
 
void tfcase_setStatus(void)
{
  srTestCaseSetStatus(srTEST_CASE_DEFAULT, srTEST_PASS, 0);
}
 
#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_setStatus)
#endif


srTestCaseSetStatusEx

The srTestCaseSetStatusEx() routine is used to set the result of test case and allow specification of an extendedFailureCode.

srWORD srTestCaseSetStatus(srTestCaseHandle_t tTestCase, srTestStatus_e eStatus, srDWORD dwDuration, srLONG lExtendedFailureCode)


Parameters Type Description
tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
eStatus Input Result of the test.
dwDuration Input The duration of the test in clock ticks. Using "0" allows the Stride Runtime (Intercept Module) to set the time automatically.
lExtendedFailureCode Input The Stride framework uses the extendedFailureCode to capture the numeric results of test method when the test method fails via a numeric (non-void, nonbool) return type.


Return Value Description
srWORD srOK on success, srERR_HANDLE_INVALID on invalid handle.


#include <srtest.h>
 
void tfcase_setStatusEx(void)
{
  srTestCaseSetStatusEx(srTEST_CASE_DEFAULT, srTEST_FAIL, 0, -5);
}
 
#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_setStatusEx)
#endif


srTestCaseAddAnnotation

The srTestCaseAddAnnotation() routine is used to add a new annotation to the specified test case.

srTestAnnotationHandle_t srTestCaseAddAnnotation(rTestCaseHandle_t tTestCase, srTestAnnotationLevel_e eLevel, const srCHAR * szName, const srCHAR * szFmtDescr, ...)


Parameters Type Description
tTestCase Input Handle to the parent test case to which new test annotation is to be added. srTEST_CASE_DEFAULT can be used for the default test case.
eLevel Input Annotation level.
  • srTEST_ANNOTATION_LEVEL_TRACE,
  • srTEST_ANNOTATION_LEVEL_DEBUG,
  • srTEST_ANNOTATION_LEVEL_INFO,
  • srTEST_ANNOTATION_LEVEL_WARN,
  • srTEST_ANNOTATION_LEVEL_ERROR,
  • srTEST_ANNOTATION_LEVEL_FATAL
szName Input Pointer to a null-terminated string representing the name of the new annotation. If null, the default host naming scheme will be used.
szFmtDescr Input Pointer to a null-terminated string representing the description of the new annotation. Cannot be null.
... Input (Optional) Variable argument list to format szFmtDescr.


Return Value Description
srTestAnnotationHandle_t Handle of the newly created test annotation. srTEST_ANNOTATION_INVALID indicates failure to create a test annotation.


#include <srtest.h>
 
void tfcase_addAnnotation(void) 
{
  for (int count = 0; count < 5; ++count)
  {
    char szName[25];
    sprintf(szName, "annotation %d", count);
    srTestAnnotationHandle_t annot = 
                     srTestCaseAddAnnotation(srTEST_CASE_DEFAULT,
                                             srTEST_ANNOTATION_LEVEL_ERROR,
                                             szName,
                                             "description of annotation");
  }
}
 
#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_addAnnotation)
#endif


srTestCaseSetData

The srTestCaseSetData() routine is used to associate a custom name|value pair with a test case.

srWORD srTestCaseSetData(srTestCaseHandle_t tTestCase, const srCHAR * szName, const srCHAR * szFmtValue, ...);


Parameters Type Description
tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
szName Input Pointer to a null-terminated string representing the name of the custom pair. Cannot be null.
szFmtValue Input Pointer to a null-terminated string representing the value of the custom pair. Cannot be null.
... Input (Optional) Variable argument list to format szFmtValue.


Return Value Description
srWORD srOK on success, srERR_HANDLE_INVALID on invalid handle.


#include <srtest.h>
 
void tfcase_setData(void)
{
  srTestCaseSetData(srTEST_CASE_DEFAULT, "MyName", "my value");
}
 
#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_setData)
#endif


srTestAnnotationAddComment

The srTestAnnotationAddComment() routine is used to add a comment (aka a log) to a test annotation.

srWORD srTestAnnotationAddComment(srTestAnnotationHandle_t tTestAnnotation, const srCHAR * szLabel, const srCHAR * szFmtComment, ...)


Parameters Type Description
tTestAnnotation Input Handle to a test annotation.
szLabel Input Pointer to a null-terminated string for the label. If null, the default label will be applied.
szFmtComment Input Pointer to a null-terminated string representing the new comment. Cannot be null.
... Input (Optional) Variable argument list to format szFmtComment.


Return Value Description
srWORD srOK on success, srERR on null string or invalid formatted string, srERR_HANDLE_INVALID on invalid handle, srERR_STR_TRUNCATED on truncated string.


#include <srtest.h>
 
void tfsuiteAnnotation_addComment(void) 
{
  srTestAnnotationHandle_t annot = 
                           srTestSuiteAddAnnotation(srTEST_ANNOTATION_LEVEL_ERROR,
                                                    "annot",
                                                    "annot description");
  srTestAnnotationAddComment(annot,
                             srNULL,
                             "this comment should print %s", "A STRING");
}
 
#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuiteAnnotation_addComment)
#endif


C++ Test Classes

The Runtime Test Services C APIs work equally well from C test functions and C++ test classes. If, however, you are using C++ it might be more convinient for you to derive your C++ test classes from the Runtime Test Services C++ base class, srTest. That way you will have access to a set of simpler to use C++ classes.

The following C++ classes are provided:


class srTest

The srTest class provides one member object testCase of class srTestCase and the folowing methods:


method GetParam

This overload of the GetParam() method is used to get the value of a named parameter as a string.

static srWORD GetParam(const srCHAR * name, srCHAR * value, srDWORD size, const srCHAR * defval = srNULL)


Parameters Type Description
name Input Pointer to a null-terminated string that represents the name of the parameter. Cannot be null.
value Output Pointer to a block of memory to store the value of the parameter with a maximum size of size chars
size Input Size in chars of the buffer pointed to by value
defvalue Input (optional) Pointer to a null-terminated string that represents a default value in case the parameter is not specified. By setting this value to srNULL (or omitting it), you can use GetParam() to test for the existence of a named parameter. If the named parameter doesn't exist, the function will return srERR_HANDLE_INVALID.


Return Value Description
srWORD srOK on success, error otherwise.


#include <srtest.h>
 
class srtest_class : public stride::srTest
{
public:
  void getParam()
  {
    GetParam("ParamName", szValue, MAX_VALUE_LEN, "default value");
  }
};
 
#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif


method GetParam

This overload of the GetParam() method is used to get the value of a named parameter as a long.

static long GetParam(const srCHAR * name, srLONG defval = 0)


Parameters Type Description
name Input Pointer to a null-terminated string that represents the name of the parameter.
defvalue Input Default value that is returned if the parameter is not found.


Return Value Description
srLONG Parameter value, or value specified by defvalue if parameter isn't found.


#include <srtest.h>
 
class srtest_class : public stride::srTest
{
public:
  void getParam()
  {
    srLONG lValue = GetParam("ParamName", -1);
  }
};
 
#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif


method GetParam

This overload of the GetParam() method is used to get the value of a named parameter as a double.

static double GetParam(const srCHAR * name, const double& defval)


Parameters Type Description
name Input Pointer to a null-terminated string that represents the name of the parameter.
defvalue Input Default value that is returned if the parameter is not found.


Return Value Description
double Parameter value, or value specified by defvalue if parameter isn't found.


#include <srtest.h>
 
class srtest_class : public stride::srTest
{
public:
  void getParam()
  {
    double dbVal = GetParam("ParamName",  0.5772156649);
  }
};
 
#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif


method AddCase

The AddCase method is used to add a new test case to the test suite.

srTestCase AddCase(const srCHAR * name, const srCHAR * descr = srNULL)


Parameters Type Description
name Input Pointer to null-terminated string that represents the name of the new test case. If empty, the default host naming scheme will be used.
descr Input (Optional) Pointer to null-terminated string representing the description of the new test case. If empty, description will be blank.


Return Value Description
srTestCase Newly created test case instance.


Example

#include <srtest.h>
#include <sstream>
 
class srtest_class : public stride::srTest
{
public:
  void suiteAddSuite()
  {
    const std::string prefix("dynamic test case ");
    for (int count = 0; count < 5; ++count)
    {
      std::stringstream strm;
      strm << prefix << count;
      stride::srTestCase tc = AddCase(strm.str().c_str());
      tc.SetStatus(srTEST_PASS);
    }
  }
};
 
#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif


method AddAnnotation

The AddAnnotation method is used to add a new test annotation to the test suite.

srTestAnnotation AddAnnotation(srTestAnnotationLevel_e level, const srCHAR * name, const srCHAR * descr, ...)


Parameters Type Description
level Input Annotation level.
  • srTEST_ANNOTATION_LEVEL_TRACE,
  • srTEST_ANNOTATION_LEVEL_DEBUG,
  • srTEST_ANNOTATION_LEVEL_INFO,
  • srTEST_ANNOTATION_LEVEL_WARN,
  • srTEST_ANNOTATION_LEVEL_ERROR,
  • srTEST_ANNOTATION_LEVEL_FATAL
name Input Pointer to null-terminated string that represents the name of the new annotation. If empty, the default host naming scheme will be used.
descr Input Pointer to null-terminated string representing the description of the new annotation. Cannot be null.


Return Value Description
srTestAnnotation Newly created annotation instance.


#include <srtest.h>
#include <sstream>
 
class srtest_class : public stride::srTest
{
public:
  void suiteAddAnnotation()
  {
    for (int count = 0; count < 5; ++count)
    {
      std::stringstream strmName;
      std::stringstream strmDescr;
      strmName << "annotation " << count;
      strmDescr << "description of annotation " << count;
      stride::srTestAnnotation ta = AddAnnotation(srTEST_ANNOTATION_LEVEL_INFO,
                                                  strmName.str().c_str(),
                                                  strmDescr.str().c_str());
    }
  }
};
 
#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif


method SetData

The SetData method is used to associate a custom name|value pair with the test suite.

srWORD SetData(const srCHAR * name, const srCHAR * value, ...)


Parameters Type Description
name Input Pointer to a null-terminated string representing the name of the custom pair. Cannot be null.
value Input Pointer to a null-terminated string representing the value of the custom pair. Cannot be null.


Return Value Description
srWORD srOK on success, srERR_HANDLE_INVALID on invalid handle.


Example

#include <srtest.h>
#include <sstream>
 
class srtest_class : public stride::srTest
{
public:
  void suiteSetData()
  {
      SetData("MyName", "my value");
  }
};
 
#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif


class srTestCase

The srTestCase class provides the following methods:

method SetStatus

The SetStatus method is used to set the result of the default test case.

srWORD SetStatus(srTestStatus_e status, srDWORD duration = 0)


Parameters Type Description
status Input Result of the test. Possible values are:
  • srTEST_FAIL
  • srTEST_PASS
  • srTEST_NOTINUSE
  • srTEST_INPROGRESS
  • srTEST_DONE - applicable to dynamic cases - sets the status to pass unless already set to fail or not-in-use
duration Input (Optional) The duration of the test in clock ticks. The default is 0.


Return Value Description
srWORD srOK on success, srERR_HANDLE_INVALID on invalid handle.


#include <srtest.h>
 
class srtest_class : public stride::srTest
{
public:
  void caseSetStatus()
  {
    testCase.SetStatus(srTEST_NOTINUSE);
  }
};
 
#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif


method AddAnnotation

The AddAnnotation method is used to add a new test annotation to the test case.

srTestAnnotation AddAnnotation(srTestAnnotationLevel_e level, const srCHAR * name, const srCHAR * descr, ...)


Parameters Type Description
level Input Annotation level.
  • srTEST_ANNOTATION_LEVEL_TRACE,
  • srTEST_ANNOTATION_LEVEL_DEBUG,
  • srTEST_ANNOTATION_LEVEL_INFO,
  • srTEST_ANNOTATION_LEVEL_WARN,
  • srTEST_ANNOTATION_LEVEL_ERROR,
  • srTEST_ANNOTATION_LEVEL_FATAL
name Input Pointer to null-terminated string that represents the name of the new annotation. If null, the default host naming scheme will be used.
descr Input Pointer to null-terminated string representing the description of the new annotation. Cannot be null.


Return Value Description
srTestAnnotation Newly created annotation instance.


#include <srtest.h>
#include <sstream>
 
class srtest_class : public stride::srTest
{
public:
  void caseAddAnnotation()
  {
    for (int count = 0; count < 5; ++count)
    {
      std::stringstream strmName;
      std::stringstream strmDescr;
      strmName << "annotation " << count;
      strmDescr << "description of annotation " << count;
      stride::srTestAnnotation ta = 
                 testCase.AddAnnotation(srTEST_ANNOTATION_LEVEL_INFO,
                                         strmName.str().c_str(),
                                         strmDescr.str().c_str());
    }
  }
};
 
#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif


method SetData

The SetData method is used to associate a custom name|value pair with the test case.

srWORD SetData(const srCHAR * name, const srCHAR * value, ...)


Parameters Type Description
name Input Pointer to a null-terminated string representing the name of the custom pair. Cannot be null.
value Input Pointer to a null-terminated string representing the value of the custom pair. Cannot be null.


Return Value Description
srWORD srOK on success, srERR_HANDLE_INVALID on invalid handle.


#include <srtest.h>
 
class srtest_class : public stride::srTest
{
public:
  void caseSetData()
  {
    testCase.SetData("MyName", "my value");
  }
};
 
#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif


class srTestAnnotation

The srTestAnnotation class provides the following methods:


method AddComment

The AddComment method is used to add a comment to the test annotation.

srWORD AddComment(const srCHAR * label, const srCHAR * comment, ...)


Parameters Type Description
label Input Pointer to null-terminated string for the label. If null, the default label will be applied.
comment Input Pointer to null-terminated string representing the new comment. Cannot be empty.


Return Value Description
srWORD srOK on success, srERR on null string or invalid formatted string, srERR_HANDLE_INVALID on invalid handle, srERR_STR_TRUNCATED on truncated string.


#include <srtest.h>
 
class srtest_class : public stride::srTest
{
public:
  void suiteAnnotationAddComment()
  {
    stride::srTestAnnotation ta = 
                 testSuite.AddAnnotation(srTEST_ANNOTATION_LEVEL_INFO,
                                         "annot",
                                         "annot description");
    ta.AddComment("this comment on annotation should print %s", "A STRING");
  }
};
 
#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif