Difference between revisions of "Studio:Reporting Sample"

From STRIDE Wiki
Jump to: navigation, search
(→‎Report Organization)
(→‎Prerequisites)
 
(15 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
==Introduction==
 
==Introduction==
−
 
+
The following content relates to the sample files and workspaces installed in ''%STRIDE_DIR%\Samples\TestScripts\Reporting''.  This sample consists of test scripts and a STRIDE workspace for sample code execution.
−
The following content relates to the sample files and workspaces installed in ''%STRIDE_DIR%\Samples\Scripting\Reporter''.  This sample consists of a [http://en.wikipedia.org/wiki/Microsoft_Visual_Studio Visual Studio] workspace for building an [[Windows_Off-Target_Apps| off-target simulator]], sample source code and test scripts, and a STRIDE workspace for sample code execution.
 
  
 
==Prerequisites==
 
==Prerequisites==
−
Before starting, some prerequisites must be satisfied. For details and instructions, see [[Host_Installation|Host Installation]].
+
Before starting, some prerequisites must be satisfied. For details and instructions, see [[Studio:Desktop Installation]].
  
 
Specifically:
 
Specifically:
 
* STRIDE must be installed on the Windows PC to be used in the training, and any required licenses must be present.
 
* STRIDE must be installed on the Windows PC to be used in the training, and any required licenses must be present.
−
* Active State Perl must be installed, and [[Host_Installation#Perl|additional perl packages]] must be downloaded and installed. '''Note:''' when the sample workspace is executed, a script runs that verifies that the required Perl packages have been installed correctly.
+
* Active State Perl must be installed, and [[Studio:Desktop Installation#Perl|additional perl packages]] must be downloaded and installed. '''Note:''' when the sample workspace is executed, a script runs that verifies that the required Perl packages have been installed correctly.
−
* Microsoft Visual Studio 2005 or a later version must be installed. If you do not currently have any version of Visual Studio, we recommend that you install the current free version of [http://en.wikipedia.org/wiki/Visual_Studio_Express Visual Studio Express].
 
  
−
==Getting Started==
+
==Overview==
−
 
+
===Getting Started===
−
To begin, open the Visual Studio Solution file in the sample directory.  This solution (and corresponding project) were created for Visual Studio 2005.  If you have a later version of Visual Studio installed, you should be able to open this solution and it will be automatically upgraded if necessary.
+
The sample is executed by opening the workspace and executing the scripts by pressing the [[Image:RunAll.png]] button. When execution is complete, the results are summarized to the Studio output window and a link to the results file is provided.  You can now peruse the output report and the scripts that generated the results.
−
 
 
−
Once you have successfully opened the solution, rebuild it. The build process has custom STRIDE build rules integrated and will produce a STRIDE database, intercept module source files, and an off-target simulator application that incorporates the sample source code.
 
−
 
 
−
Once the build is complete, perform the following steps to run the sample scripts in the workspace:
 
−
 
 
−
# launch the off-target simulator, Reporter_Sample.exe.  This will run in a standard console window.
 
−
# open a command prompt window and change to this sample's directory.
 
−
# at the command prompt, run the command <tt>'''WorkspaceRun.pl -x setup -x teardown'''</tt>.  This will execute all of the sample test scripts in the workspace and displays the results. You may open a browser and display the report written to the Reporter_Sample.html file in the sample's directory .
 
−
# quit the Reporter_Sample.exe application by typing 'q' in its console window.
 
  
−
==Overview==
+
===Reporter Basics===
 
The STRIDE [[Reporter]] object provides a means to organize and collect test results data, then publish results in a format of your choosing. The STRIDE Studio application integrates a reporter object with its collection of test scripts, and this sample's Studio workspace defines the hierarchy of the test scripts that will be the resulting hierarchy of the test suites and test cases in the generated report. The following describes the contents of the test report in more detail.
 
The STRIDE [[Reporter]] object provides a means to organize and collect test results data, then publish results in a format of your choosing. The STRIDE Studio application integrates a reporter object with its collection of test scripts, and this sample's Studio workspace defines the hierarchy of the test scripts that will be the resulting hierarchy of the test suites and test cases in the generated report. The following describes the contents of the test report in more detail.
  
−
==Report Organization==
+
====Report Organization====
−
 
 
 
As stated earlier, the '''Reporter''' stores test results hierarchically. At the top level, it holds a collection of one or more child '''[[Reporter#reporter.Suites|Suites]]'''. These Suites--in turn--hold a collection of zero or more child '''[[Reporter#reporter.Suites.Item.Tests|Tests]]''', a second collection of zero or more Suites, and a third collection of zero or more '''[[Reporter#reporter.Suites.Item.Annotations|Annotations]]'''.
 
As stated earlier, the '''Reporter''' stores test results hierarchically. At the top level, it holds a collection of one or more child '''[[Reporter#reporter.Suites|Suites]]'''. These Suites--in turn--hold a collection of zero or more child '''[[Reporter#reporter.Suites.Item.Tests|Tests]]''', a second collection of zero or more Suites, and a third collection of zero or more '''[[Reporter#reporter.Suites.Item.Annotations|Annotations]]'''.
  
Line 37: Line 25:
 
[[Image:ReporterHierarchy.jpg|frame|right|Reporter Object Hierarchy]]
 
[[Image:ReporterHierarchy.jpg|frame|right|Reporter Object Hierarchy]]
  
−
=== The Suite Object ===
+
==== The Suite Object ====
 
The [[Reporter#reporter.Suites.Item|Suite]] object is used to group related Tests (and Annotations--if present) so that their results can be summarized on an output report.
 
The [[Reporter#reporter.Suites.Item|Suite]] object is used to group related Tests (and Annotations--if present) so that their results can be summarized on an output report.
  
Line 45: Line 33:
 
* Description
 
* Description
  
−
 
+
==== The Test Object ====
−
=== The Test Object ===
 
 
The [[Reporter#reporter.Suites.Item.Tests.Item|Test]] object represents a specific test that is run. The result of a test is reflected in its Status property (''"PASS"/"FAIL"'').
 
The [[Reporter#reporter.Suites.Item.Tests.Item|Test]] object represents a specific test that is run. The result of a test is reflected in its Status property (''"PASS"/"FAIL"'').
  
Line 55: Line 42:
 
* Status
 
* Status
  
−
=== The Comment Object ===
+
==== The Comment Object ====
 
The [[Reporter#reporter.Suites.Item.Tests.Item.Comments.Item|Comment]] object allows you to attach a text comment to a Test or Annotation. Comments are useful for logging test script flow and other test information.
 
The [[Reporter#reporter.Suites.Item.Tests.Item.Comments.Item|Comment]] object allows you to attach a text comment to a Test or Annotation. Comments are useful for logging test script flow and other test information.
  
Line 63: Line 50:
 
* Label
 
* Label
  
−
=== The Annotation Object ===
+
==== The Annotation Object ====
 
The [[Reporter#reporter.Suites.Item.Annotations.Item|Annotation]] object is used when you want to associate a file or a URI with a Suite. (e.g. a screen shot or other data file).
 
The [[Reporter#reporter.Suites.Item.Annotations.Item|Annotation]] object is used when you want to associate a file or a URI with a Suite. (e.g. a screen shot or other data file).
  
Line 69: Line 56:
  
 
==Sample Scripts==
 
==Sample Scripts==
−
The scripts that demonstrate the use of the reporter object are in the workspace '''test''' folder, in the '''Files''' tab. Examples scripts in JScript ('''NegativeTests.js''') and Perl ('''PositiveTests.pl''') are provided. Each uses the sample test function '''IsLeapYear''', passing in integral values and using the return value to create the content of the test suite. The scripts use the AutoScript objects Functions collection to call the test function (see the [[AutoScript_Samples]] article to learn more about using the AutoScript object with scripts.)
+
We provide a range of examples in JScript and perl that demonstrate how to use the reporter object model.  The scripts in both languages are functionally identical, differing only in syntax. The following sections briefly describe each broad category. We encourage you to read and understand the provided examples.
 +
 
 +
=== Suites ===
 +
 
 +
==== add ====
 +
Creates new [[Reporter#reporter.Suites.Item|Suite]] objects in a [[Reporter#reporter.Suites|Suites]] collection.
 +
 
 +
==== remove ====
 +
Removes Suite objects from a collection by name and index. Also demonstrates use of [[Reporter#suites-removeall|RemoveAll]].
  
−
The following describes the highlights of how the Reporter object is used in the sample scripts.
+
==== find ====
 +
Gets instances of existing Suite objects in a collection using the [[Reporter#suites-item|Item]] and [[Reporter#suites-find|Find]] methods. Also shows how the Parent property can be used to navigate upwards in the Suite hierarchy.
  
−
=== Creating Suites ===
+
==== output ====
−
Suites can be created from a script, but it's unnecessary when running within STRIDE Studio, as a Suite is created for you automatically for each folder under '''Script Files''' in the workspace tree.
+
Persists results for a Suite as HTML and XML.  The XML output is also used to reload the results.
  
−
For convenience, the automatically-created Suite object that is a script's immediate parent is injected into the script's namespace. This Suite is accessed in the sample scripts by using the pre-defined '''testSuite''' object.
+
==== custom data ====
 +
Creates [[Reporter#reporter.Suites.Item.CustomData.Item|Custom Data]] fields for a Suite.
  
−
=== Creating Tests ===
+
==== annotations ====
−
The Test objects are created and added to the parent Suite from within each script. A typical pattern is to implement one test per script, but you can create as many Tests as you like in a single script.
+
Creates [[Reporter#reporter.Suites.Item.Annotations.Item|Annotations]] for a Suite.
  
−
==== perl ====
+
=== Tests ===
−
<pre>
 
−
# create a new test in the parent suite
 
−
my $test = $main::testSuite->Tests->Add("Leap Year Positive Tests");
 
−
$test->{Description} = "Tests the IsLeapYear functionality";
 
−
</pre>
 
  
−
==== JScript ====
+
==== status ====
−
<pre>
+
Creates several [[Reporter#reporter.Suites.Item.Tests.Item|Tests]] and shows various means of settings status.
−
// create a new test in the parent suite
 
−
var test = testSuite.Tests.Add("Leap Year Negative Tests");
 
−
test.Description = "Tests the IsLeapYear functionality";
 
−
</pre>
 
  
−
=== Creating Comments ===
+
==== remove ====
 +
Removes Test objects from a collection by name and index. Also demonstrates use of [[Reporter#tests-removeall|RemoveAll]].
  
−
==== perl ====
+
==== find ====
−
<pre>
+
Gets instances of existing Test objects in a collection using the [[Reporter#tests-item|Item]] method. Also shows how the Parent property can be used to navigate upwards in the Test/Suite hierarchy.
−
# add a new comment to the test report with the passed-in label
 
−
my $comment = $test->Comments->Add("Testing $input[$i]");
 
−
 
−
...
 
−
 
−
# test the return value; note that the value is returned as the name of the enumerator
 
−
if ($f->ReturnValue eq "YES") {
 
−
    # set the comment text
 
−
    $comment->{Comment} = "Pass";
 
−
}
 
−
else {
 
−
    # set the comment text
 
−
    $comment->{Comment} = "Fail";
 
−
    # indicate that at least one test failed
 
−
    $testResult = 1;
 
−
}
 
−
</pre>
 
−
==== JScript ====
 
−
<pre>
 
−
// add a new comment to the test report with the passed-in label
 
−
var comment = test.Comments.Add("Testing " + arInput[nIndex]);
 
  
−
...
+
==== comments ====
 +
Creates [[Reporter#reporter.Suites.Item.Tests.Item.Comments.Item|Comments]] for a Test.
  
−
// test the return value; note that the value is returned as the name of the enumerator
+
==== metrics ====
−
if (f.ReturnValue == "NO") {
+
Demonstrates the use of implicit and explicit duration data.  Also shows how to add optional Test Metric data.
−
    comment.Comment = "Pass";
 
−
}
 
−
else {
 
−
    comment.Comment = "Fail";
 
−
    testResult = false;
 
−
}
 
−
</pre>
 
  
−
=== Setting Test Status ===
+
=== Dual Mode ===
 +
We provide one script in each language that is written so as to be runnable both within and outside of stride studio (standalone).  These scripts are executed as part of the normal workspace execution.  In order to run the scripts outside studio, follow these steps:
  
−
==== perl ====
+
<ul>
 +
<li>Exit STRIDE Studio</li>
 +
<li>Open a DOS command prompt and change to the ''%STRIDE_DIR%\Samples\Scripting\Reporting'' directory</li>
 +
<li>To execute the perl standalone script:
 
<pre>
 
<pre>
−
# set the overall test status
+
  perl Scripts\perl\DualMode\standalone.pl
−
if ($testResult == 0) {
 
−
    $test->{Status} = "PASS";
 
−
}
 
−
else {
 
−
    $test->{Status} = "FAIL";
 
−
}
 
 
</pre>
 
</pre>
−
==== JScript ====
+
To execute the JScript version:
 
<pre>
 
<pre>
−
// set the overall test status
+
  cscript Scripts\jscript\DualMode\standalone.js
−
test.Status = testResult ? "PASS" : "FAIL";
 
 
</pre>
 
</pre>
−
 
+
Each script will produce a corresponding report file in the current directory.
−
==Test Execution==
+
</li>
−
 
+
</ul>
−
This sample demonstrates two different techniques for executing the sample test scripts.
+
[[Category:Studio:Samples]]
−
 
 
−
===Command Line Execution===
 
−
 
 
−
Command line execution for sample STRIDE test workspace is done using the [[WorkspaceRun.pl|WorkspaceRun utility]]. Here is an example of specific syntax to execute a test workspace. All of these commands can be invoked from a standard [http://en.wikipedia.org/wiki/Command_Prompt_(Windows) command shell] (or other shell of your choosing) and the arguments shown assume that the commands are executed with the sample's directory as the starting directory. You must have your Reporter_Sample.exe application running in order for the runner to be able to initiate a connection to the target simulator.
 
−
 
 
−
  WorkspaceRun.pl -x setup -x teardown
 
−
 
 
−
When you run this command, you should see console output like:
 
−
 
 
−
  Opening workspace C:\S2\Seaside\Samples\Scripting\Reporter/Reporter_Sample.ssw..
 
−
  excluding folder setup in workspace C:\S2\Seaside\Samples\Scripting\Reporter/Reporter_Sample.ssw
 
−
  running folder test in workspace C:\S2\Seaside\Samples\Scripting\Reporter/Reporter_Sample.ssw
 
−
  excluding folder teardown in workspace C:\S2\Seaside\Samples\Scripting\Reporter/Reporter_Sample.ssw
 
−
  Test results written to C:\S2\Seaside\Samples\Scripting\Reporter\Reporter_Sample.html
 
−
  ***************************************************************************
 
−
  Results Summary
 
−
  ***************************************************************************
 
−
    Passed:              1
 
−
    Failed:              1
 
−
    In Progress:          0
 
−
    Not Applicable:      0
 
−
    ...in 2 suites.
 
−
  ***************************************************************************
 
−
 
 
−
===Workspace-Based Execution===
 
−
 
 
−
After building the target application (described [[#Getting_Started|above]]), open the sample STRIDE workspace (.ssw file) and activate the '''Files''' tab in the workspace panel. Under the '''Script Files''' folder, the '''setup''' and '''teardown''' folders provide basic infrastructure scripts that start and stop the target application (Reporter_Sample.exe). The scripts that drive the testing are in the workspace '''test''' folder, as described in the [[#Sample_Scripts|Sample Scripts]] section above. The test function is defined in the ''LeapYear.h'' header file under '''Source Files'''.<br>
 
−
To run the workspace, click on the '''Run All''' button on the Studio application toolbar. [[Image:run_all.gif]]<br>
 
−
After the workspace scripts have executed, you should see the following message on the Output panel:
 
−
 
 
−
  Test Report created <font color="blue"><u>file://C:\S2\Seaside\Samples\Scripting\Reporter\Reporter_Sample.html</u></font>
 
−
 
−
Double-clicking on the hyperlink will launch the test results report in your browser.
 
−
 
 
−
[[Category: Samples]]
 

Latest revision as of 23:01, 6 May 2010

Introduction

The following content relates to the sample files and workspaces installed in %STRIDE_DIR%\Samples\TestScripts\Reporting. This sample consists of test scripts and a STRIDE workspace for sample code execution.

Prerequisites

Before starting, some prerequisites must be satisfied. For details and instructions, see Studio:Desktop Installation.

Specifically:

  • STRIDE must be installed on the Windows PC to be used in the training, and any required licenses must be present.
  • Active State Perl must be installed, and additional perl packages must be downloaded and installed. Note: when the sample workspace is executed, a script runs that verifies that the required Perl packages have been installed correctly.

Overview

Getting Started

The sample is executed by opening the workspace and executing the scripts by pressing the RunAll.png button. When execution is complete, the results are summarized to the Studio output window and a link to the results file is provided. You can now peruse the output report and the scripts that generated the results.

Reporter Basics

The STRIDE Reporter object provides a means to organize and collect test results data, then publish results in a format of your choosing. The STRIDE Studio application integrates a reporter object with its collection of test scripts, and this sample's Studio workspace defines the hierarchy of the test scripts that will be the resulting hierarchy of the test suites and test cases in the generated report. The following describes the contents of the test report in more detail.

Report Organization

As stated earlier, the Reporter stores test results hierarchically. At the top level, it holds a collection of one or more child Suites. These Suites--in turn--hold a collection of zero or more child Tests, a second collection of zero or more Suites, and a third collection of zero or more Annotations.

Each Test has a collection of zero or more Comments, and each Annotation has a collection of zero or more Comments.

The diagram at the right shows an example hierarchy.

Reporter Object Hierarchy

The Suite Object

The Suite object is used to group related Tests (and Annotations--if present) so that their results can be summarized on an output report.

The Suite object exposes a set of properties, the most important are listed below.

  • Name
  • Description

The Test Object

The Test object represents a specific test that is run. The result of a test is reflected in its Status property ("PASS"/"FAIL").

The most important properties of the Test object are listed below.

  • Name
  • Description
  • Status

The Comment Object

The Comment object allows you to attach a text comment to a Test or Annotation. Comments are useful for logging test script flow and other test information.

The Comment object exposes two properties listed below.

  • Comment
  • Label

The Annotation Object

The Annotation object is used when you want to associate a file or a URI with a Suite. (e.g. a screen shot or other data file).

The Annotation object is designed for use with STRIDE Portal, and will not be covered further here. For more information, see the Annotation item section of the Reporter reference article.

Sample Scripts

We provide a range of examples in JScript and perl that demonstrate how to use the reporter object model. The scripts in both languages are functionally identical, differing only in syntax. The following sections briefly describe each broad category. We encourage you to read and understand the provided examples.

Suites

add

Creates new Suite objects in a Suites collection.

remove

Removes Suite objects from a collection by name and index. Also demonstrates use of RemoveAll.

find

Gets instances of existing Suite objects in a collection using the Item and Find methods. Also shows how the Parent property can be used to navigate upwards in the Suite hierarchy.

output

Persists results for a Suite as HTML and XML. The XML output is also used to reload the results.

custom data

Creates Custom Data fields for a Suite.

annotations

Creates Annotations for a Suite.

Tests

status

Creates several Tests and shows various means of settings status.

remove

Removes Test objects from a collection by name and index. Also demonstrates use of RemoveAll.

find

Gets instances of existing Test objects in a collection using the Item method. Also shows how the Parent property can be used to navigate upwards in the Test/Suite hierarchy.

comments

Creates Comments for a Test.

metrics

Demonstrates the use of implicit and explicit duration data. Also shows how to add optional Test Metric data.

Dual Mode

We provide one script in each language that is written so as to be runnable both within and outside of stride studio (standalone). These scripts are executed as part of the normal workspace execution. In order to run the scripts outside studio, follow these steps:

  • Exit STRIDE Studio
  • Open a DOS command prompt and change to the %STRIDE_DIR%\Samples\Scripting\Reporting directory
  • To execute the perl standalone script: 
      perl Scripts\perl\DualMode\standalone.pl

    To execute the JScript version: 

      cscript Scripts\jscript\DualMode\standalone.js

    Each script will produce a corresponding report file in the current directory.

Retrieved from "https://www.stridewiki.com/index.php?title=Studio:Reporting_Sample&oldid=12268"