Difference between revisions of "Running Tests (old)"

From STRIDE Wiki
Jump to: navigation, search
Line 21: Line 21:
 
By default, tests are run in the order in which their corresponding SCL pragmas were seen by the [[s2scompile|STRIDE compiler]] (This can be overridden; see below.)
 
By default, tests are run in the order in which their corresponding SCL pragmas were seen by the [[s2scompile|STRIDE compiler]] (This can be overridden; see below.)
  
−
== Examples ==
+
= Examples =
  
−
=== Running Diagnostics ===
+
== Running STRIDE Diagnostics ==
 
S2 Technologies supplies a set of diagnostic tests that can be used to verify correct operation.
 
S2 Technologies supplies a set of diagnostic tests that can be used to verify correct operation.
  
−
By building in the files srdiag.h and srdiag.c (located in the SDK/Runtime directory of each SDK), these tests become available.
+
By building in the files srdiag.h and srdiag.c (located in the SDK/Runtime directory of each SDK) to your target application, these tests become available.
  
 
Assuming that:
 
Assuming that:
Line 35: Line 35:
 
  stride --database TestApp.sidb --device TCP:localhost:8000 --diagnostics
 
  stride --database TestApp.sidb --device TCP:localhost:8000 --diagnostics
  
−
==== Console Output ====
+
=== Console Output ===
 
Information is written to the console while the test progresses (note that the link tests can take ~30 seconds), and a results summary is written at the conclusion of the test.
 
Information is written to the console while the test progresses (note that the link tests can take ~30 seconds), and a results summary is written at the conclusion of the test.
  
Line 53: Line 53:
 
</pre>
 
</pre>
  
−
==== Detailed Output ====
+
=== Detailed Output ===
 
Detailed results output are also created and written to the local disk.
 
Detailed results output are also created and written to the local disk.
  
Line 60: Line 60:
 
Additionally, a companion .xsl file is downloaded and written alongside the xml file if you are connected to the Internet. If you open the xml file in a browser, the xsl stylesheet will be automatically applied to render html to the browser.
 
Additionally, a companion .xsl file is downloaded and written alongside the xml file if you are connected to the Internet. If you open the xml file in a browser, the xsl stylesheet will be automatically applied to render html to the browser.
  
−
=== Desktop Interactive Testing ===
+
== Desktop Test Development ==
 
Useful for 'pre-flight' desktop testing as well as target debugging, the following command line runs the target tests without publishing results to [[STRIDE Test Space]].
 
Useful for 'pre-flight' desktop testing as well as target debugging, the following command line runs the target tests without publishing results to [[STRIDE Test Space]].
  
−
stride -r --database "/my/path/stride.sidb" --device TCP:localhost:8000
+
We assume the following:
 +
* We are testing against a target build that contains Test Units that I don't want to run. (Perhaps they test code that I am not responsible for.)
 +
* The target device is running, and is available on the LAN via TCP/IP at 192.168.0.53
 +
* I want to run only Test Units named myTest1, myTest2, and myTest3
  
−
=== Testing with Publication to Test Space ===
+
The command line to run the desired tests would be: 
 +
 
 +
stride -r "myTest1,myTest2,myTest3" --database "/my/path/targetApp.sidb" --device TCP:192.168.0.53:8000
 +
 
 +
A couple of things to note:
 +
* If others may be running tests using referenced database, you should specify a unique output file location (via the -o option) to prevent overwriting the output file at the default location.
 +
* If you will be running many tests against a specific database and/or device, consider setting [[Stride_Executable#Environment_Variables|environment variables]] to the corresponding values, or put command line options into an options file (via the -O option).
 +
* You can refer to the device address using a DNS name if desired (e.g. TCP:testcomputer:8000)
 +
 
 +
== Desktop Testing with Publication to Test Space ==
 
This example will run the target tests and then publish the results to [[STRIDE Test Space]] according to the configuration specified in your [[Test Spaces Configuration File]].
 
This example will run the target tests and then publish the results to [[STRIDE Test Space]] according to the configuration specified in your [[Test Spaces Configuration File]].
  
 
  TestUnitRun.pl -d stride.db -u -n="My Test Run"
 
  TestUnitRun.pl -d stride.db -u -n="My Test Run"
  
 +
== Continuous Integration Testing ==
  
−
=== Specifying Tests to Run ===
+
TBD
−
By default all tests are run in the order in which their corresponding SCL pragmas were seen by the [[s2scompile|STRIDE compiler]]. This is typically what you want, but you may also specify which test suites to run and in what order they are to be run.<ref> A ''test suite'' here is the grouping of tests by SCL pragma; this corresponds to either:
 
−
* a test class
 
−
** '''#pragma scl_test_class()'''
 
−
* a set of free functions, or
 
−
** '''#pragma scl_test_flist()'''
 
−
* a C-based class)
 
−
** '''#pragma scl_test_cclass()'''
 
−
</ref>
 
−
 
 
−
* Specify test(s) to run and their order on the [[TestUnitRun.pl]] command line (using -r)
 
−
* Specify test(s) to run and their order by means of an [[Test Unit Order File | Order File]]
 
  
 +
<hr/>
 
=== Notes ===
 
=== Notes ===
 
<references/>
 
<references/>
  
 
[[Category:Test Units]]
 
[[Category:Test Units]]

Revision as of 17:19, 28 May 2009

Target-based Test Units are controlled by a host computer, physically connected to the target via a configurable communication channel (TCP/IP or serial port). This article describes some common usage patterns and presents several examples.

Prerequisites

In order to run target-based Test Units, the following prerequisites are required.

Host

Target

  • Target transport configured
    • TCP/IP over port 8000 is the default target transport. To apply other configurations, see Host Transport
  • Target built with STRIDE library built in and one or more test cases
  • Target app running [1]

Preparing to Run Your Tests

Tests are executed on the target by running the program stride on the host computer. See the linked article for reference information.

In all cases,

  • The stride executable must be able to read the STRIDE database (.sidb) file that corresponds to the target build

By default, tests are run in the order in which their corresponding SCL pragmas were seen by the STRIDE compiler (This can be overridden; see below.)

Examples

Running STRIDE Diagnostics

S2 Technologies supplies a set of diagnostic tests that can be used to verify correct operation.

By building in the files srdiag.h and srdiag.c (located in the SDK/Runtime directory of each SDK) to your target application, these tests become available.

Assuming that:

  • your STRIDE database is named TestApp.sidb and resides in the current directory, and
  • the corresponding target application is running on the host computer

the command line to run diagnostic tests is: 

stride --database TestApp.sidb --device TCP:localhost:8000 --diagnostics

Console Output

Information is written to the console while the test progresses (note that the link tests can take ~30 seconds), and a results summary is written at the conclusion of the test. 

Loading database...
Connecting to device...
Diagnosing target application...
  runtime version: 4.1.01
  link
  payload fragmentation
  runtime configuration
  stub-proxy deadlock
  target characteristics
Summary: 64 passed, 0 failed, 0 in progress, 0 not in use.
Disconnecting from device...
Saving result file...

Detailed Output

Detailed results output are also created and written to the local disk.

Since we did not specify an output file to stride above, the xml output is written by default to the directory containing the designated STRIDE database. The name of the file is taken from the database file name with the .sidb replaced by .xml.

Additionally, a companion .xsl file is downloaded and written alongside the xml file if you are connected to the Internet. If you open the xml file in a browser, the xsl stylesheet will be automatically applied to render html to the browser.

Desktop Test Development

Useful for 'pre-flight' desktop testing as well as target debugging, the following command line runs the target tests without publishing results to STRIDE Test Space.

We assume the following:

  • We are testing against a target build that contains Test Units that I don't want to run. (Perhaps they test code that I am not responsible for.)
  • The target device is running, and is available on the LAN via TCP/IP at 192.168.0.53
  • I want to run only Test Units named myTest1, myTest2, and myTest3

The command line to run the desired tests would be: 

stride -r "myTest1,myTest2,myTest3" --database "/my/path/targetApp.sidb" --device TCP:192.168.0.53:8000

A couple of things to note:

  • If others may be running tests using referenced database, you should specify a unique output file location (via the -o option) to prevent overwriting the output file at the default location.
  • If you will be running many tests against a specific database and/or device, consider setting environment variables to the corresponding values, or put command line options into an options file (via the -O option).
  • You can refer to the device address using a DNS name if desired (e.g. TCP:testcomputer:8000)

Desktop Testing with Publication to Test Space

This example will run the target tests and then publish the results to STRIDE Test Space according to the configuration specified in your Test Spaces Configuration File. 

TestUnitRun.pl -d stride.db -u -n="My Test Run"

Continuous Integration Testing

TBD

Notes

  1. ↑ If target is multi-process, both the target and the STRIDE I/O daemon process must be started.
Retrieved from "https://www.stridewiki.com/index.php?title=Running_Tests_(old)&oldid=9746"