Difference between revisions of "Frequently Asked Questions About STRIDE"
(→What are the advantages of Test Points over logging?) |
(→Are all Test Points active?) |
||
| Line 32: | Line 32: | ||
=== Are all Test Points active? === | === Are all Test Points active? === | ||
| − | No. Under normal testing scenarios, only the specific test points that are needed for a test are actually broadcast through the system. We accomplish this by setting test point filters (by label) on the system whenever one of the Test Point setup functions is called (in script or native code). These filters are reset or removed at the end of the test, so in general ''none'' of the test points are actually sent through the system if no test is currently active. That said, there are a few special use cases in which ''all'' test points become active in the system - namely, when [[Tracing | tracing]] is activated on the host runner and when a specific test case uses a set that include | + | No. Under normal testing scenarios, only the specific test points that are needed for a test are actually broadcast through the system. We accomplish this by setting test point filters (by label) on the system whenever one of the Test Point setup functions is called (in script or native code). These filters are reset or removed at the end of the test, so in general ''none'' of the test points are actually sent through the system if no test is currently active. That said, there are a few special use cases in which ''all'' test points become active in the system - namely, when [[Tracing | tracing]] is activated on the host runner and when a specific test case uses a set that include ''TEST_POINT_EVERYTHING_ELSE''. In general, however, the test points that are actually sent from the system are ''only'' those that are needed to execute the behavior validation for the current test. |
=== Will it affect performance? === | === Will it affect performance? === | ||
Revision as of 11:39, 16 June 2010
Contents
Integration with STRIDE
What is the size of the STRIDE Runtime?
The STRIDE Runtime is a source package that supports connectivity with the host system and provides services for testing and source instrumentation. The runtime is tailored specifically to embedded applications, overhead is minimal. It consumes very little memory for table and control block storage. Resource usage is configurable and can be tailored to the limitations of the target platform.
| Aspect | Resources |
|---|---|
| Code Space | About 90-130 KB depending on the compiler of use and the level of optimization. |
| Memory Usage | Configurable, by default set to about 10 KB |
| Threads | 3 Threads; configurable priority; blocked when inactive |
What is the processing overhead?
The STRIDE Runtime overhead is minimized by collecting raw data on the target, and transferring information at a low priority task to the host for processing. Processing is only active when executing tests via the STRIDE Runner.
Source Instrumentation
What are the advantages of Test Points over logging?
Test Points can be used for direct validation during real-time execution. Logging systems, while often useful for historical record, can only be used for post-processing validation at best. What's more, since there is no standard way to post-process logs, users often rely on manual inspection or non-scalable homegrown solutions for validation. Test Point validation is fully automated using the STRIDE Framework, which provides robust harnessing and reporting. Test Points also can include optional data payloads which can be very useful for validating state when a point is hit. Test Points have potentially lower impact on the system than standard logging since they (1) are only sent from the origin if a test is actively subscribed to test point (labels are used as a filter at runtime) and (2) test points can be completely disabled (no-opped) in your build by undefining a single preprocessor macro (STRIDE_ENABLED).
What about source instrumentation bloat?
Any mature code base has some level of diagnostic bloat to it. This often takes the form of ad-hoc logging or debug statements. With STRIDE Test Points and Test Logs, you open your software to better automated test scenarios. All STRIDE instrumentation takes the form of single line macros, so the amount of instrumentation bloat is no worse than other typical ad-hoc diagnostics. What's more, the STRIDE macros are all designed to be quickly no-opped in a build via a single preprocessor macro, making it possible to completely eliminate any actual impact on certain builds, if so desired.
Are all Test Points active?
No. Under normal testing scenarios, only the specific test points that are needed for a test are actually broadcast through the system. We accomplish this by setting test point filters (by label) on the system whenever one of the Test Point setup functions is called (in script or native code). These filters are reset or removed at the end of the test, so in general none of the test points are actually sent through the system if no test is currently active. That said, there are a few special use cases in which all test points become active in the system - namely, when tracing is activated on the host runner and when a specific test case uses a set that include TEST_POINT_EVERYTHING_ELSE. In general, however, the test points that are actually sent from the system are only those that are needed to execute the behavior validation for the current test.
Will it affect performance?
Our experience on a wide-range of systems has shown minimal impact from the STRIDE instrumentation. The STRIDE Runtime has been designed to be small, portable, and readily configurable, allowing it to be optimized for the platform's specific characteristics.
Should I leave Test Points in?
Yes. Once you have some behavior tests written, it's worthwhile to maintain that instrumentation and the corresponding tests, which allows you to run the tests on any stride-enabled build. All of the instrumentation macros are easily no-opped via a single preprocessor flag, so you can choose to effectively remove the instrumentation code on select builds (production/release builds, for example). The ultimate value of instrumentation is the continuous feedback you get by regularly executing the automated tests on the build.
Testing
What languages are supported?
Unit tests and API Tests are written using C and C++. Behavior tests can be written using Perl and C/C++.
Can I test using my host computer?
Yes. Tests can be implemented and executing using the Off-Target Environment.
Test Automation
What is continuous integration and why should I care?
The key principle of continuous integration is regular testing of your software--ideally done in an automated fashion. STRIDE tests are reusable and automated. Over time, these tests accumulate, providing more and more comprehensive coverage. By automating the execution of tests and results publication via STRIDE Test Space with every software build, development teams gain immediate feedback on defects and the health of their software. By detecting and repairing defects immediately, the expense and time involved with correcting bugs is minimized.
Does STRIDE support continuous integration?
Yes. The STRIDE Runner provides a straightforward means to connect to the device under test and execute the test cases you've implemented using the STRIDE Framework. The runner allows you to configure which tests to run and how to organize the results using subsuites in the report. Since the runner supports an option-based command line interface, this tool is easy to integrate with typical continuous integration servers. Please refer to this article as well.
Where/how do you store test results?
When you execute your tests using the STRIDE Runner, upon completion the results are written to an xml file. This xml file uses a custom schema for representing the hierarchy of results (suites, cases, etc.). These files also include a stylsheet specification (which will be written to the same directory as the xml file) that allows them to be viewed as HTML in a browser. You are free to store these files for future use/reference. The runner also supports direct uploading to STRIDE Test Space which is a hosted web application for viewing and collaborating on your test results. Once you are regularly executing tests, whether automatically or manually, we recommend you use the upload feature to persist and share your test results.
Can I get email containing test reports?
Yes. If you use STRIDE Test Space to store your results, you can optionally configure your test space(s) to automatically notify users when new results are uploaded. The email that is generated by Test Space contains only summary information and provides links so that you can view the complete report data.
If you are using a continuous integration server to initiate your testing, it's likely that it supports different forms of notification when the testing is complete, so it's often possible to attach the xml report data as part of the CI server notification.
Getting Started
How long does it take to install STRIDE?
For standard platforms such as Linux the installation process varies between a few hours to a couple of days. We provide SDK packages that work out of the box for our Off-Target Environment, but also provide a reference to integrators.
For proprietary targets a Platform Abstraction Layer (PAL) is required. The PAL provides the glue between the STRIDE Runtime and services offered by the OS. It is the only piece of the runtime that is customized between operating systems. The implementation of the PAL ranges between a day to a week depending on the complexity of the OS. There is also a build integration step. This involves integrating the STRIDE Build Tools into your software make process. The activity ranges from a single day to several.
What kind of training is required?
Our training approach is based on wiki articles, samples, and leveraging the Off-Target Environment. The training have been setup for self-guided instructions that can be leverage for an initial introduction of the technology and on-demand for specific topics when required.
