Many popular test tools use graphical approaches to specify signal-based tests. These, for instance, are based on direct graphical manipulation of visualized signal curves or visualized state automata. Usually, these can only be created, changed and visualized using the tools of the test software vendor because of the underlying proprietary file format.
This article addresses the benefits of text-based test specifications compared with purely graphical approaches and finally highlights how Arttest combines the best of both in a single tool with maximal benefit for its users.
1 PROS AND CONS OF GRAPHICAL APPROACHES
With the term “graphical test specification”, we relate to approaches, where entire or parts of tests are specified using a visual editor, such as the one shown below.
These graphical editors enable the user to manipulate signal data by adding, and removing signal segments with different patterns to the signal curves. Segments can usually be parameterized, e.g., directly by manipulating the values or via drag-and-drop operations, which provide visual live feedback, a major benefit compared to non-visual specifications.
Another advantage of visual specifications is that mouse operations are, in general, faster performed compared to keyboard interactions as many studies from UI researchers have shown in the past.
However, many research on comparison between keyboard and mouse interactions focus on menu and button interaction, where it suffices to hit an area of the screen but no exact point. With regard to graphical signal specifications, start, end and parameters, e.g. slope, number of repetitions or other values, hitting an exact location is desirable. For instance, if the input to a system under test shall be a step from 0 to 10, drag & drop operations to specify an exact step height of 10 may be difficult, resulting in values around 10, e.g. 10.002 etc. Since for testing purposes it is often essential to use very specific values and thus keyboard interaction is required nonetheless.
Depending on the tool and test environment, functionality varies. For instance, some tools enable graphical signal specification for input stimuli only while others also support the specification of expected outputs, often along with tolerances.
Besides the graphical signal editor, these tools have in common, that tests are stored in a proprietary format. This complicates compatibility with versioning systems such as Git, SVN and others since these operate on the basis of line-wise diffs, which is only available for textual descriptions. Moreover, test modification and signal data inspection are only possible using the tools of the test vendor.
The functionality available to edit and modify signal curves graphically varies heavily between tools. This has direct impact on the flexibility and efficiency to create, adapt and reuse specified signals. For instance, most editors and tools consider signals to be unrelated to each other. Thus, changes impacting multiple signals, e.g. shifting the already specified signals in time as shown in the figure below, often requires manual changes on multiple signals. Moreover, the stored and modeled information is the signal curve itself, which might make it difficult for others to understand how multiple signals relate to each other and why they were modeled the way they are for specific tests.
Another example for often-unsupported operation is Copy & Paste on arbitrary signal parts, e.g., copying the behavior of all signals between seconds 5 and 9 to second 10 to 14 in the figure above. Thus, complex changes and adaptions affecting entire tests are error prone and cumbersome.
Moreover, graphical signal specifications do not allow including reactive elements, such as conditions or events as part of the signal curves. Thus, if such functionality is supported at all by the test environment, editors detached from the signal curve specification need to be used and thus the relation to the signal curves is lost.
2 PROS AND CONS OF TEXTUAL APPROACHES AND FORMATS
Textual test and signal descriptions can solve many of the drawbacks for graphical signal specifications discussed before. Operations being rather complex on graphical level where signals are considered unrelated to each other, such as Copy & Paste or shifts, are easily performed on textual basis by insertion of corresponding specification parts.
Moreover, the use of specific languages close to wording and syntax of requirements and/or natural language eases the understanding and derivation of tests from requirements. In addition, semantical explanations and reasoning for test creation can be included as comments. Thus, reuse and adaption by others is easier.
Since no proprietary format is used, the text is readable by humans, can be changed by a variety of editors and processed by versioning control systems such as SVN, GIT etc. with full support of version diffs, branching and merging.
Considering the drawbacks of textual specifications, it is obvious, that there is some overhead due to the need to type syntactically and semantically correct test definitions without visual feedback about the signal curves and data. This makes purely unsupported textual specifications as shown below more flexible but also more error prone and difficult to handle.
3 HOW ARTTEST MAKES TEXTUAL APPROACHES SUPERIOR
With Arttest, Mindmotiv provides a solution based on well-structured textual signal specifications overcoming the drawbacks of textual specifications compared to graphical approaches.
The textual language is close to natural language and thus easy to understand, learn and remember. Tests are structured such that stimuli signals and expected behavior with potential tolerances are separated but based on identical language constructs.
Due to the textual nature of the specification, benefits as introduced above, including compatibility with versioning systems, possibility to add comments at arbitrary places, reuse and Copy&Paste operations or shifting groups of signals, apply.
However, all the drawbacks of purely textual specifications are eliminated due to tool support, providing a superior specification solution.
Based on an analysis of the underlying system-under-test and its interface, Arttest checks whether the textual specification complies with the data gathered. For instance, used signal names are checked for existence and boundaries of data types and compliance with signal ranges from the system under test are checked. Moreover, compliance with the specification structure is ensured and syntax checks for the specification language are performed. All checks are live and feedback is directly shown during test creation.
To improve readability and speed up test creation, syntax highlighting, auto formatting and content completion based on system data is available. Thus, the tester does not need to know the language syntax by heart nor type complete signal names or language elements. Using shortcuts and content completion can significantly reduce the number of key presses required.
By providing a graphical live-preview on the textually specified signal changes in a separate window, Arttest adds the ultimate benefit of visually displaying the described signal curves to the textual specification. Changes in the text are directly processed and the plots updated to represent an always-up-to-date view on specified signal curves. To improve the overview, plotted signals can be arbitrarily grouped, enabled/disabled and colored, and the plot supports zoom operations to focus on specific parts of the signals. In summary, the plot provides all benefits of a graphical signal specification without the drawbacks of a purely graphical specification, introduced before.
4 CONCLUSION
Comparing textual and graphical signal specification solutions, both have their benefits and drawbacks. Graphical signal specification provides immediate and intuitive feedback on the changes being made at the cost of flexibility and compatibility with regard to capturing semantic explanations, versioning systems, available functionality and lock-in effects on required tools. Purely textual specifications, however, require knowledge about the specification language, are sensitive to syntactical errors and lack visual feedback. Arttest combines both specification approaches being based on a powerful textual specification language, which provides all benefits of text based approaches but eliminates the drawbacks due to heavy tool support. Together with the live signal curve visualization, this paves the road for effective yet flexible test specification. Have you have worked with either kind of specification methods or are you curious about trying the best of both worlds? We would be delighted to hear about your experiences. Get in touch now.