|
|
| (75 intermediate revisions by 9 users not shown) |
| Line 1: |
Line 1: |
| | == Introduction == | | == Introduction == |
| − | The Test FList Samples are part of [[Test_Units_Training#Introduction|STRIDE Test Units Training]]. FList is an abbreviation for 'Function List' used in both pragmas and documentation. The Test FList Samples pertain to test units that contain lists of functions to be executed. Unlike the Test Class and Test CClass test units, the Test FList test units were not created specifically for C or C++. However, they are predominantly used to call test functions (which for now means no class member functions).
| |
| − | <br><br>
| |
| − | The following content relates to the sample files and workspaces installed in ''%STRIDE_DIR%\Samples\TestUnits\TestFList''. This sample consists of a [http://en.wikipedia.org/wiki/Microsoft_Visual_Studio Visual Studio] workspace for building an [[Windows_Off-Target_Apps| off-target simulator]], sample [[Test Units|test flist]] source code, and a STRIDE workspace for doing more advanced test flist execution.
| |
| | | | |
| − | == Getting Started ==
| + | Function Lists are abbreviated as ''flist'' in both pragmas (as in [[scl_test_flist]]) and documentation. The Test flist Samples pertain to test units that contain lists of functions to be executed. The Test FList functionality is intended as a simple grouping of test functions and are designed to be used with the C language (although it is not restricted from compilation in a C++ environment as well). Test FLists does not support more advanced usages patterns like private test data. If you need more advanced functionality, consider using [[Test Class Sample|Test Classes (C++)]] or [[Test CClass Sample|Test C-Classes]]. |
| − | To begin, open the Visual Studio Solution file in the sample directory. This solution (and corresponding project) were created for Visual Studio 2005. If you have a later version installed, you should be able to open this solution (it will be automatically upgraded if necessary). If you do not have any version of Visual Studio, we recommend you install the current free version of [http://en.wikipedia.org/wiki/Visual_Studio_Express Visual Studio Express].
| |
| | | | |
| − | Once you have successfully opened the solution, rebuild it. The build process has custom STRIDE build rules integrated, The rebuilding will produce a STRIDE database, intercept module source files, and an off-target simulator application that incorporates the test class source.
| + | == Tests Description == |
| | | | |
| − | Once the build is complete, perform the following steps to run the test flists in the workspace:
| + | === Basic === |
| | + | These examples cover the simplest, easiest way to code STRIDE scl_test_flist functionality. These examples use simple [http://en.wikipedia.org/wiki/Plain_Old_Data_Structures POD] return types to indicate status and do not annotate the tests with any rich information (such as comments). |
| | | | |
| − | # launch the off-target simulator, TestFList.exe. This will run in a standard console window.
| + | ==== basic_simple ==== |
| − | # open a command prompt window and change to this sample's directory.
| + | This example demonstrates passing and failing tests using the primary integer (int, short, and char) types that infer status from. |
| − | # at the command prompt, run the command <tt>'''TestUnitRun.pl -v'''</tt>. This will execute all of the test units in the workspace and open a browser to display the results.
| |
| − | # quit the TestFList.exe application by typing 'q' in its console window.
| |
| | | | |
| − | == Sample Test FLists == | + | ==== basic_fixtures ==== |
| − | Now that you have built the off-target simulator and executed the test flists it contains, you can take time to peruse the test flist source and the corresponding results that each produces. This section provides a brief description for each.
| + | This example shows how to use [[Test Unit Pragmas#Fixturing Pragmas|setup and teardown]] fixtures. The setup and teardown methods are called immediately before and after the execution of each test method, respectively. |
| | | | |
| − | === Basic Examples === | + | === Runtime Services=== |
| − | These examples cover the simplest, easiest way to code a STRIDE test flist. These examples use simple [http://en.wikipedia.org/wiki/Plain_Old_Data_Structures POD] return types to indicate status and do not annotate the tests with any rich information (such as comments). | + | These examples cover basic usage of our [[Runtime Test Services|Runtime Test Services API]] (as declared in srtest.h). |
| | | | |
| − | ==== 01_01_Basic_Simple ==== | + | ==== runtimeservices_simple ==== |
| − | Demonstrates passing and failing tests using the four primary POD types that we can infer status from (int, bool, short, and char).
| + | This example shows how to use [[Runtime Test Services#srTestCaseSetStatus|srTestCaseSetStatus]] to set status and [[Runtime Test Services#srTestCaseAddAnnotation|srTestCaseAddAnnotation]] to add a comment. |
| | | | |
| − | ==== 01_02_Basic_Fixtures ==== | + | ==== runtimeservices_dynamic ==== |
| − | Shows how to use [[Test_Units#Pragmas_for_Test_Units|setup and teardown]] fixtures. The setup and teardown methods are called immediately before and after the execution of each test method, respectively.
| + | This example shows how to use [[Runtime Test Services#srTestSuiteAddCase|srTestSuiteAddCase]], [[Runtime Test Services#srTestCaseAddAnnotation|srTestCaseAddAnnotation]], and [[Runtime Test Services#srTestAnnotationAddComment|srTestAnnotationAddComment]] for dynamic suite, test, and annotation creation in the context of a single test method. |
| | | | |
| − | ==== 01_03_Basic_Exceptions ==== | + | ==== runtimeservices_override ==== |
| − | Demonstrates how exceptions thrown from a test method are caught by our intercept module and noted in the results. Any exception that is caught by the harness is assumed to indicate failure.
| + | This example shows how to use [[Runtime Test Services#srTestCaseSetStatus|srTestCaseSetStatus]] to override the status that would otherwise be inferred from the return value. |
| | | | |
| − | === Runtime Services Examples === | + | ==== runtimeservices_varcomment ==== |
| − | These examples cover basic usage of our Runtime Test Services API (as declared in srtest.h).
| + | This example demonstrates the use of [http://en.wikipedia.org/wiki/Printf printf] style format strings with [[Runtime Test Services#srTestCaseAddAnnotation|srTestCaseAddAnnotation]]. |
| | | | |
| − | ==== 02_01_RuntimeServices_Simple ==== | + | == Run Tests == |
| − | Shows how to use [[Test_Units#srTestCaseSetStatus|srTestCaseSetStatus]] to set status, [[Test_Units#srTestCaseAddComment|srTestCaseAddComment]] to add a comment, and srTEST_ADD_COMMENT_WITH_LOCATION to add a comment that automatically includes line and file information.
| + | Now launch the test app (if you have not already) and execute the runner with the following commands: |
| | | | |
| − | ==== 02_02_RuntimeServices_Dynamic ==== | + | ''Test FList tests'': |
| − | Shows how to use [[Test_Units#srTestSuiteAddSuite|srTestSuiteAddSuite]] and [[Test_Units#srTestSuiteAddTest|srTestSuiteAddTest]] for dynamic suite and test creation in the context of a single test method.
| + | <pre> |
| | + | stride --device="TCP:localhost:8000" --database="../out/TestApp.sidb" --run="s2_testflist_basic_fixtures; s2_testflist_basic_simple" --output=FList.xml |
| | + | </pre> |
| | | | |
| − | ==== 02_03_RuntimeServices_Override ====
| + | Note the command will produce distinct result files for the run (per the --output command above). Please use the result file to peruse the results by opening each in your browser. |
| − | Shows how to use [[Test_Units#srTestCaseSetStatus|srTestCaseSetStatus]] to override the status that would otherwise be inferred from the return value.
| |
| | | | |
| − | ==== 02_04_RuntimeServices_VarComment ==== | + | == Observations == |
| − | Demonstrates the use of [http://en.wikipedia.org/wiki/Printf printf] style format strings with [[Test_Units#srTestCaseAddComment|srTestCaseAddComment]].
| + | This sample demonstrates a simpler packaging technique that is appropriate for systems that support C compilation only (no C++). [[Test_Units#Test_Units|FLists]] are simple a collection of functions that are called in sequence. There is no shared state or data, unless you arrange to use global data for this purpose. Review the source code in the directory and follow the sample description. |
| | | | |
| − | === srTest Examples ===
| + | The following are some test observations: |
| − | These examples show to how to use the [[Test_Units#C.2B.2B_Test_FLists|stride::srTest]] base class for your test classes. When you publicly inherit from srTest, you get access to default testCase and testSuite members and their associated methods.
| |
| | | | |
| − | ==== 03_01_srTest_Simple ====
| + | * flist tests support setup/teardown fixturing, but '''not''' parameterization or exception handling. |
| − | Demonstrates the use of [[Test_Units#SetStatus|testCase.setStatus]] to set the status for test cases.
| + | * we've again provided documentation using doxygen formatting for these samples. However, because there is no storage-class entity with which the docs are associated in an FList, there are some restrictions to the documentation, which you can read about [[Test_API#Test_FLists|here]]. |
| | + | * notice how the [[Scl_test_flist|scl_test_flist pragma]] requires you to both create a name for the test unit (first argument) '''and''' explicitly list each test method that is part of the unit. This is one disadvantage of flist over a test class (the latter does not require explicit listing of each test since all conforming public methods are assumed to be test methods). |
| | | | |
| − | ==== 03_02_srTest_Dynamic ====
| |
| − | Shows how to use [[Test_Units#AddSuite|testSuite.AddSuite]] and [[Test_Units#AddTest|testSuite.AddTest]] for dynamic suite and test case creation within the context of one test method.
| |
| − |
| |
| − | == Test FList Execution ==
| |
| − | This sample demonstrates two different techniques for executing test classes.
| |
| − |
| |
| − | === Commandline Execution ===
| |
| − | Command line execution for test flists is done using the [[Test_Runners#TestUnitRun.pl|TestUnitRun utility]]. Here are several examples of specific syntax to execute test flists. All of these commands can be invoked from a standard command shell and the arguments shown assume that the commands are executed with the sample's directory as the starting directory. You must have your TestFList.exe application running in order for the runner to be able to initiate a connection to the target simulator. In addition, you should verify that your %STRIDE_DIR%\bin\transport.cfg file is using the TCP transport to connect to port 8000 (these are the default settings when the product is installed).
| |
| − |
| |
| − | ==== Simple execution of all test units ====
| |
| − | The following command executes all of the test units found in the STRIDE database you have previously generated. For the purpose of this sample, since there is only one database, the -d parameter is not strictly needed, but we show it here for completeness.
| |
| − |
| |
| − | TestUnitRun.pl -d TestFList.sidb
| |
| − |
| |
| − | This command executes all Test Units found in the database in descending alpha-numeric sort order. Any Test FList initialization arguments are given default values (typically zero or NULL).
| |
| − |
| |
| − | When you run this command, you should see console output like:
| |
| − |
| |
| − | Attempting connection using [Sockets (S2)] transport ...
| |
| − | Connected to device.
| |
| − | Initializing STRIDE database objects...
| |
| − | Done.
| |
| − | Running Test Basic::Exceptions...
| |
| − | Running Test Basic::Fixtures...
| |
| − | Running Test Basic::Simple...
| |
| − | Running Test RuntimeServices::Dynamic...
| |
| − | Running Test RuntimeServices::Override...
| |
| − | Running Test RuntimeServices::Simple...
| |
| − | Running Test RuntimeServices::VarComment...
| |
| − | Running Test srTest::Dynamic...
| |
| − | Running Test srTest::Simple...
| |
| − | Disconnected from device.
| |
| − | Test Results saved to C:\s2\seaside\Samples\TestUnits\TestFList\TestFList.xml
| |
| − | Test Report saved to C:\s2\seaside\Samples\TestUnits\TestFList\TestFList.html
| |
| − | ***************************************************************************
| |
| − | Results Summary
| |
| − | ***************************************************************************
| |
| − | Passed: 28
| |
| − | Failed: 12
| |
| − | In Progress: 2
| |
| − | Not Applicable: 2
| |
| − | ...in 12 suites.
| |
| − | ***************************************************************************
| |
| − |
| |
| − | ==== Execution based on an order file ====
| |
| − | TestUnitRun can optionally base its execution on simple text file input. We have provided a simple order file, ''RunSimple.txt'', which specifies a subset of all the Test FLists in this sample. This order file also shows how to create subsuites in the final output by using the special '''{suitepath}''' syntax, as described in [[Test_Runners#Usage|the usage section]].
| |
| − |
| |
| − | TestUnitRun.pl -d TestFList.sidb -o RunSimple.txt
| |
| − |
| |
| − | ...and will produce this output:
| |
| − |
| |
| − | Attempting connection using [Sockets (S2)] transport ...
| |
| − | Connected to device.
| |
| − | Initializing STRIDE database objects...
| |
| − | Done.
| |
| − | Running Test Basic::Simple...
| |
| − | Running Test RuntimeServices::Simple...
| |
| − | Running Test srTest::Simple...
| |
| − | Disconnected from device.
| |
| − | Test Results saved to C:\s2\seaside\Samples\TestUnits\TestFList\TestFList.xml
| |
| − | Test Report saved to C:\s2\seaside\Samples\TestUnits\TestFList\TestFList.html
| |
| − | ***************************************************************************
| |
| − | Results Summary
| |
| − | ***************************************************************************
| |
| − | Passed: 13
| |
| − | Failed: 6
| |
| − | In Progress: 2
| |
| − | Not Applicable: 2
| |
| − | ...in 6 suites.
| |
| − | ***************************************************************************
| |
| − |
| |
| − | ==== Execution based on file system order ====
| |
| − | TestUnitRun can also try to infer execution order and suite hierarchy from filesystem organization. If you have organized your test flist source files (only the files that contain the scl_test_flist pragma matter here) in a filesystem hierarchy that you want to mimic in your tests, you can specify a root of a directory tree that contains the relevant test flist source. TestUnitRun will walk the directory structure and determine order and hierarchy of tests based on the directory structure. To see an example of this in action, you can execute this command with the sample:
| |
| − |
| |
| − | TestUnitRun.pl -d TestFList.sidb -f Tests
| |
| − |
| |
| − | This will cause the runner to examine the Tests subdirectory structure and build a hierarchy of tests based on that directory tree. Subdirectories will map to suites in the final report. Here is the output for this example:
| |
| − |
| |
| − | Attempting connection using [Sockets (S2)] transport ...
| |
| − | Connected to device.
| |
| − | Initializing STRIDE database objects...
| |
| − | Done.
| |
| − | Running Test Basic::Simple...
| |
| − | Running Test Basic::Fixtures...
| |
| − | Running Test Basic::Exceptions...
| |
| − | Running Test RuntimeServices::Simple...
| |
| − | Running Test RuntimeServices::Dynamic...
| |
| − | Running Test RuntimeServices::Override...
| |
| − | Running Test RuntimeServices::VarComment...
| |
| − | Running Test srTest::Simple...
| |
| − | Running Test srTest::Dynamic...
| |
| − | Disconnected from device.
| |
| − | Test Results saved to C:\s2\seaside\Samples\TestUnits\TestFList\TestFList.xml
| |
| − | Test Report saved to C:\s2\seaside\Samples\TestUnits\TestFList\TestFList.html
| |
| − | ***************************************************************************
| |
| − | Results Summary
| |
| − | ***************************************************************************
| |
| − | Passed: 28
| |
| − | Failed: 12
| |
| − | In Progress: 2
| |
| − | Not Applicable: 2
| |
| − | ...in 15 suites.
| |
| − | ***************************************************************************
| |
| − |
| |
| − |
| |
| − | === Workspace-based Execution ===
| |
| − | We also provide a sample STRIDE workspace that demonstrates the use of script execution with STRIDE Studio to manage test order and hierarchy. This workspace was created using [[WorkspaceSetup.pl]] and the [[Provided_Frameworks#WindowsTestApp|WindowsTestApp framework]]. The setup and teardown folders provided basic infrastructure scripts that start/stop the simulator application (TestFList.exe) and to manage traceviews used for [[Runtime_Reference#Logging_Services|srPrint]] message collection. The scripts that drive the testing are in the workspace '''test''' folder. What follows is a brief description for each.
| |
| − |
| |
| − | ==== RunAll ====
| |
| − | This folder contains a single script that iterates through the entire collection of test units and executes them one at a time. The order of execution will be in descending alpabetical order (on name) since we explictly called the [[Ascript#ascript.TestUnits|ArrangeBy]] collection method.
| |
| − |
| |
| − | ==== Run Individual ====
| |
| − | This folder shows how to use individual scripts to execute test classes. Each script takes the very simple form:
| |
| − |
| |
| − | ascript.TestUnits.Item('''TEST_FLIST_NAME''').Run();
| |
| − |
| |
| − | ..where '''TEST_FLIST_NAME''' is the name of the test flist you want to run. You can then use the Studio tree control to change the order and hierarchy of each item by moving the scripts and creating folders. The sample contains individual scripts for a few of the sample test flists - you are free to move, add, or delete any items as you experiment with the workspace.
| |
| | | | |
| | [[Category: Samples]] | | [[Category: Samples]] |
Introduction
Function Lists are abbreviated as flist in both pragmas (as in scl_test_flist) and documentation. The Test flist Samples pertain to test units that contain lists of functions to be executed. The Test FList functionality is intended as a simple grouping of test functions and are designed to be used with the C language (although it is not restricted from compilation in a C++ environment as well). Test FLists does not support more advanced usages patterns like private test data. If you need more advanced functionality, consider using Test Classes (C++) or Test C-Classes.
Tests Description
Basic
These examples cover the simplest, easiest way to code STRIDE scl_test_flist functionality. These examples use simple POD return types to indicate status and do not annotate the tests with any rich information (such as comments).
basic_simple
This example demonstrates passing and failing tests using the primary integer (int, short, and char) types that infer status from.
basic_fixtures
This example shows how to use setup and teardown fixtures. The setup and teardown methods are called immediately before and after the execution of each test method, respectively.
Runtime Services
These examples cover basic usage of our Runtime Test Services API (as declared in srtest.h).
runtimeservices_simple
This example shows how to use srTestCaseSetStatus to set status and srTestCaseAddAnnotation to add a comment.
runtimeservices_dynamic
This example shows how to use srTestSuiteAddCase, srTestCaseAddAnnotation, and srTestAnnotationAddComment for dynamic suite, test, and annotation creation in the context of a single test method.
runtimeservices_override
This example shows how to use srTestCaseSetStatus to override the status that would otherwise be inferred from the return value.
This example demonstrates the use of printf style format strings with srTestCaseAddAnnotation.
Run Tests
Now launch the test app (if you have not already) and execute the runner with the following commands:
Test FList tests:
stride --device="TCP:localhost:8000" --database="../out/TestApp.sidb" --run="s2_testflist_basic_fixtures; s2_testflist_basic_simple" --output=FList.xml
Note the command will produce distinct result files for the run (per the --output command above). Please use the result file to peruse the results by opening each in your browser.
Observations
This sample demonstrates a simpler packaging technique that is appropriate for systems that support C compilation only (no C++). FLists are simple a collection of functions that are called in sequence. There is no shared state or data, unless you arrange to use global data for this purpose. Review the source code in the directory and follow the sample description.
The following are some test observations:
- flist tests support setup/teardown fixturing, but not parameterization or exception handling.
- we've again provided documentation using doxygen formatting for these samples. However, because there is no storage-class entity with which the docs are associated in an FList, there are some restrictions to the documentation, which you can read about here.
- notice how the scl_test_flist pragma requires you to both create a name for the test unit (first argument) and explicitly list each test method that is part of the unit. This is one disadvantage of flist over a test class (the latter does not require explicit listing of each test since all conforming public methods are assumed to be test methods).
