Expectations
Stride supports Behavior Testing, which is different than Unit Testing or API Testing in that it does not focus on calling functions and validating return values. Behavior Testing validates the expected sequencing and state of the software executing under normal operating conditions. In order to begin this type of testing, you must (1) instrument the source under test with Test Points and (2) define the expectations of the Test Points for each test scenario.
The Validation Model is based on validating that the Expected List of Test Points have been hit as well as optionally validating any state data associated with a Test Point member. Through the use of an Unexpected List, it's possible to simulataneously validate that specified Test Points are NOT hit during the execution of the scenario.
The Expected List and Unexpected List declared for a specific testing scenario define the active set of Test Points for the system. All other Test Points within the software are not active and have nominal impact.
Expected List
The first step in defining Expectations is describing the set of Test Points expected to be hit during a test scenario. This is the list of Test Points and the expected number of hits (count) for each of the Test Points. Any expected state data associated with a Test Point that requires validation should be included.
The validation of the Expected List is done when any of the following conditions are met:
- The Expected List sequencing has been met completely
- The time allocated for validation has expired
- State data associated with a Test Point is declared invalid
- An out-of-sequence Test Point has been encountered (a concrete failure is identified)
- An Unexpected Test Point has been encountered
Expected TEST POINTS list:
| Label | Count | Expected State Data |
| Name 1 | 1 | <describe data payload validation requirements if applicable> |
| Name 2 | 1 + | <describe data payload validation requirements if applicable> |
| name ... | n | <...> |
Sequencing Properties
The Expected List requires defining sequencing properties used for the validation. This involves establishing how to process the ordering of the Test Points being hit, how strict to handle duplications of Test Point members, and if to continue capturing Test Points for the entire time allocated for the test scenario even though the Expected List sequencing requirements have been met.
Expected TEST POINTS sequencing properties:
| Properties | Description |
| Ordered | Test Points are expected to be hit in the exact order defined in the list |
| Unordered | Test Points can be hit in any order defined in the list |
| Strict | Test Points specified in the list must match the exact count (i.e. no duplication) |
| Non-Strict | Test Points specified in the list can be duplicated anywhere in the sequence |
| Continue | Test Points validation will continue even though the Expected List has been met so far |
State Data Validation
State data validation for a Test Point member is optional. The Expected List can associate a predicate[1] as a necessary condition for validation of a Test Point member when being processed. The predicate provides an extra level of validation (beyond sequencing) focusing on the state data associated with the Test Point.
The predicate can declare the Test Point:
- Valid - successful, thus the expected count is decremented
- Invalid - the expectations is NOT met, indicating failure
- Ignored - no affect on the validation, continue to wait on the current Test Point
Unexpected List
An Unexpected List of Test Points can optionally be defined. This list works in conjunction with the Expected List. The list defines a set of Test Points that are to be treated as failures if any one of them is encountered (hit) during a testing scenario.
This list can be made up of the complement of the Expected List, thus including all Test Points not included in that defined set. This type of setting requires enabling all Test Points defined in the software during the execution of the test.
If the Unexpected List members intersect with the Expected List members, that is considered an input error.
Special Processing
Members
There are two special members that can be used for the Expected List called ANY IN SET and ANY AT ALL. These special member values are used to customize the success criteria (i.e. trigger condition) for the entry defined within the Expected List. Both types require a predicate[1] to determine if the Test Point is valid, invalid, or ignored. The ANY IN SET refers to the set of points defined by the Expected List while the ANY AT ALL refers to all Test Points in the system. By definition the ANY AT ALL member requires enabling all Test Points defined in the software during the execution of the test.
Concerning an Unordered Expected List, only one special member can be used. It will be processed first independent of its location within the list. The Count attribute is also optionally available to be used with the special members, indicating the number of valid returns from the predicate until the next Test Point in the list is validated.
There is one special member that can be used for the Unexpected List called EVERYTHING ELSE. This special member can only be used as a single member only within an Unexpected List. It takes the complement of the set of Test Points defined within the Expected List. This requires enabling all Test Points defined in the software during the execution of the test. Also, by definition any non-expected Test Point hit during the test would be deemed a failure.
Counts
There is one special count value that can be associated with a Test Point defined in the Expected List called ANY COUNT. This special count is used to focus on capturing a variable number of Test Point hits -- 0 or more. In essences the Test Point has no consequences of the Expected List sequencing, except if a predicate[1] associated with the Test Point returns invalid.
Use Cases
The following use cases provide examples of defining Expectations based on different testing scenarios.
Sequencing Properties
Use case examples for validating the sequencing of Test Points.
Expecting an exact ordered sequence of Test Points to be hit
Expect = [A,B,C,D]; ORDERED, STRICT Hits = [A,B,C,D] - PASS Hits = [A,B,D,C] - FAIL (D hit before C)
Expecting an exact ordered of Test Points but with duplicates
Expect = [A,B,C,D]; ORDERED, STRICT Hits = [A,B,B,C,D] - FAIL (B duplicated with strict defined)
Expect = [A,B,C,D]; ORDERED, NON-STRICT Hits = [A,B,B,C,D] - PASS Hits = [A,C,B,C,D] - PASS Hits = [A,B,C,B,D] - PASS
Expecting a set of Test Points to be hit, but the order does not matter
Expect = [A,B,C,D]; UNORDERED, STRICT Hits = [B,A,C,D] - PASS Hits = [B,A,C,B,D] - FAIL (B hit twice)
Expect = [A,B,C,D]; UNORDERED, NON-STRICT Hits = [B,A,C,B,D] - PASS
Expecting a set of Test Points to be hit, but order and duplications does not matter
Expect = [A,B,C,D]; UNORDERED, NON-STRICT Hits = [B,A,C,B,D] - PASS Hits = [B,A,A,A,D] - FAIL (C never hit)
Expecting an exact ordered sequence of Test Points to be hit, but checking for any extra hits for total duration of test scenario
Expect = [A,B,C,D]; ORDERED, STRICT, CONTINUE Hits = [A,B,C,D] - PASS Hits = [A,B,C,D,A] - FAIL (A hit after valid sequence met based on test timeout)
Count Attribute
Use case examples that leverage the Count attribute associated with a list member.
Expecting a sequence of Test Points, including multiple hits for 2 members
Expect = [A,B:3,C,D:2]; ORDERED, STRICT Hits = [A,B,B,B,C,D,D] - PASS Hits = [A,B,B,C,B,D,D] - FAIL (B not hit three times before C)
Expect = [A,B:3,C,D:2]; ORDERED, NON-STRICT Hits = [A,B,B,B,C,B,D,D] - PASS
Expecting a set of Test Points being hit, including multiple hits (with duplications), but order does not matter
Expect = [A,B:3,C,D]; UNORDERED, STRICT Hits = [B,A,B,C,B,D] - PASS Hits = [B,A,B,B,C,B,D] - FAIL (B duplicated on 4th hit)
Expect = [A,B:3,C,D]; UNORDERED, NON-STRICT Hits = [B,A,B,B,C,B,D] - PASS
State Data Validation
Use case examples leveraging predicates[1] to validate state data.
Simple example showing associating a predicate with Test Point named A.
Expect = [A:p1(tp),B,C,D]; ORDERED, STRICT
p1(tp) { return VALID }
Hits = [A,B,C,D] - PASS
Expecting Test Point B to be hit 3 times
Expect = [A,B:3:p1(tp),C,D]; UNORDERED, STRICT
p1(tp) { return VALID unless 3rd time called than return INVALID}
Hits = [B,A,B,C,B,D] - FAIL (predicate returns invalid on 3rd B hit)
Special Members
Use case examples using the two special members within an Expected List.
Waiting on a single trigger (D) to start the processing of an Expected List
Expect = [ANY_IN_SET:p1(tp),A,B,C:2,D]; ORDERED, STRICT
p1(tp) {return VALID when D hit otherwise IGNORE}
Hits = [A,B,A,D*,A,B,C,C,D] - PASS
Waiting on Test Point outside of the Expected List to trigger processing
Expect = [ANY_AT_ALL:p1(tp),A,B,C:2,D]; ORDERED, STRICT
p1(tp) {return VALID when x hit}
Hits = [A,B,A,x*,A,B,C,C,D] - PASS
Waiting on a multiple triggers - 1st trigger used to start processing and 2nd used midstream
Expect = [ANY_IN_SET:p1(tp),A,B,ANY_IN_SET:p2(tp),C:2,D,E]; ORDERED, STRICT
p1(tp) {return VALID when D hit}
p2(tp) {return VALID when E hit}
Hits = [A,B,A,D*,A,B,A,A,B,E*,C,C,D,E] - PASS
Expect = [ANY_IN_SET:p1(tp),A,B,C:2,D]; UNORDERED, NON-STRICT
p1(tp) {return VALID when D hit}
Hits = [A,B,A,D*,A,A,A,D,A,C,B,C,D] - PASS
This test uses the ANY AT ALL along with a fix count of 3
Expect = [ANY_AT_ALL:p1(tp):3,A,B,C]; ORDERED, STRICT
p1(tp) {return VALID}
Hits = [A,z,x,A,B,C] - PASS
Special Counts
Use case examples leveraging the special count attributes within an Expected List.
Expecting between 0 and more Test Point As to be hit before processing the remaining members.
Expect = [A:ANY_COUNT,B,C,D]; ORDERED, STRICT Hits = [B,C,D] - PASS Hits = [A,A,B,C,D] - PASS
This test passes immediately after the A is hit (example of a weird expectation)
Expect = [A,B:ANY_COUNT]; ORDERED, STRICT Hits = [A,..] - PASS
This test also passes, but will process all Test Point B hits until the test time expires
Expect = [A,B:ANY_COUNT]; ORDERED, STRICT, CONTINUE Hits = [A,B,B,B] - PASS (the wait timeout will complete the validation) Hits = [A] - PASS
This test fails because all Test Points are checked once the test time expires
Expect = [A]; ORDERED, STRICT, CONTINUE Hits = [A,A] - FAILS (the wait timeout than will complete the validation)
This test validates a set of Test Points focusing exclusively on the associated state data
Expect = [A:p1(tp):ANY_COUNT, B:p2(tp):ANY_COUNT, C:p2(tp):ANY_COUNT]; UNORDERED, CONTINUE (STRICT NA) Hits = [A,A,B,A,C,C,C,A,..] - PASS (unless any of the predicates fails) Hits = [A,B,C,D] - PASS
Miscellaneous
The following are miscellaneous use cases.
Expecting an exact ordered sequence of Test Points to be hit, but verifying certain Test Points are NOT hit
Expect = [A,B,C,D]; ORDERED, STRICT Unexpect = [x,y] Hits = [A,B,C,D] - PASS Hits = [A,B,B,C,D] - FAIL Hits = [A,B,C,y,..] - FAIL
This test will always pass and log every Test Point hit during test time duration. Note all Test Points are active.
Expect = [ANY_AT_ALL:p1(tp):ANY_COUNT]; CONTINUE {other sequencing properties NA}
p1(tp) {return VALID}
Hits = [A,A,B,x,y,D, ..] - PASS (the wait timeout will complete the validation)
This test will log every Test Point hit during test, looking for X and Y as failures. Note all Test Points are active.
Expect = [ANY_AT_ALL:p1(tp):ANY_COUNT]; CONTINUE {other sequencing properties NA}
p1(tp) {return VALID}
Unexpect = [X,Y]
Hits = [A,A,B,x,y,D,Y] - FAIL (last Test Point hit causes a failure)
This test will fail on any Test Point that is hit. Note all Test Points are active. Test will last based on time duration or the 1st Test Point hit.
Expect = NULL; CONTINUE {other sequencing properties NA}
Unexpect = [EVERYTHING_ELSE]
Hits = [A,..] - FAIL
This test will ignore any Test Point hit during test, looking for X and Y as failures.
Expect = NULL; CONTINUE {other sequencing properties NA}
Unexpect = [X,Y]
Hits = [..,X,..] - FAIL
This test will fail on any Test Points hit except for A & B. Note all Test Points are active. Test will last until failure or timeout.
Expect = [A,B]; UNORDERED, NON-STRICT, CONTINUE Unexpect = [EVERYTHING_ELSE] Hits = [..,A,..,X] - FAIL
