The size and complexity of embedded software is growing exponentially, in particular in the automotive industry. As one consequence, the manual creation of a sufficient amount of test cases becomes more and more challenging. But the automatic generation of test cases has always been a controversial topic. While some people dream about stopping any manual test activities, others say that test generation is not allowed. But who is right?
Traditional Embedded Software Development
Let’s have a quick look at the possible test goals for which we can generate test cases:
Structural coverage goals come from the model or code directly. This includes, but is not limited to, Statement, Decision and MD/DC coverage. As a rule, it is intended that these test goals are covered by test cases. If those goals cannot be covered, it could be useful to have a closer look. It might be possible to have dead code which should be avoided in your production code.
Robustness coverage goals are also derived from the code and include, for instance, Division-by-Zeros and critical Down Casts. In contrast to the structural coverage goals, this condition is undesirable because it could have a critical impact on the robustness of your system.
Drive-To-State goals are user defined and represent specific states of the system under test (SUT). This determines confirmation of whether or not certain states can be reached, and what input combinations would create those states. This is also useful for monitoring a critical state that should not be covered by any input combination.
Requirements-based test cases can also be automatically generated, and are based on a machine-readable representation of the requirement previously created by the user. This is also called a “Formal Requirement” because it has a clearly defined syntax and semantics, and also describes timing constraints if applicable.
With these test goals in mind, we can have a look at available technologies to create these test cases. Basically, there are two technologies available on the market.
Random
The most common approach to generate test data is to generate random input data and to check what test goals are covered based on the generated data. This is done on an instrumented version of a model or code. To make this approach more efficient, a heuristic can control in what ranges random data is generated.
The advantage of this approach is that it is quite fast and it can be easily implemented. However, depending on the concrete implementation, you might get many redundancies and probably very long stimuli vectors. In addition, it can only show that a certain test goal can be covered. However, this method is unable to determine if a test goal like a Division-by-Zero can occur. This approach only makes sense for generating stimuli vectors for structural test goals.
Complete
The second approach is called Model Checking. Please note, that this method should not be confused with a guideline checker in an environment like Simulink. Model Checking originated in the early 1980s was originally invented by two research groups in the United States (Clark, Emmerson, McMillan) and France (Quielle/Sifakis). Model Checking is an efficient search procedure for finite state machines to determine whether they fulfill a specification expressed in temporal logic. To optimize this approach symbolic model checking or reduced ordered binary decision diagrams have been introduced.
A model checker performs a complete analysis of the behavior of a system against a specific static or temporal property and proves if this property holds or if it can be violated. In the case of a violation, a counter example is provided that enables the user to debug the violation. Complete analysis means that all possible runs of a system will be analyzed within one analysis task and deliver complete mathematical proof of the results.
To generate stimulus vectors or test cases with this method, the Model Checker provides a counter example when a supposedly unreachable test goal can be covered. This approach can be used for all test goals mentioned before.
Scenario-based Virtual Validation for ADAS/AD
For ADAS/AD systems (where a safety-critical software is able to partially or completely take over the control of the vehicle), new standards like SOTIF lead to the need to perform a scenario-based virtual validation of the complete system. There are even already activities in the ASAM group to create a standardized language to describe these scenarios (OpenSCENARIO).
But it is also obvious that the needed number of scenarios can’t be created manually. Recorded real-world scenarios can help, but recording the needed number of scenarios is economically infeasible.
To solve this, BTC proposes an intuitive and graphical high-level language to describe abstract traffic scenarios. The language describes an abstract scenario in multiple phases, each being modeled in a graphical way. These abstract scenarios are the basis for all subsequent steps in the validation process including automatic test generation and automatic scenario observation. The actual test cases (or concrete simulation scenarios) need to be generated with a smart strategy, on one hand avoiding a test explosion problem, on the other hand providing a statistical reasoning for completeness to enable homologation.
Despite being a proprietary language, BTC is aiming for full compatibility with the OPENScenario Standard and is therefore actively participating in the corresponding ASAM groups.
Conclusion
Automatic test generation is a powerful approach to deal with the growing complexity in embedded software projects. But the generated tests can only be as good as the input we provide to the test generator. If we don’t provide any information, we can still generate structural tests for the software unit test, which are very valuable for use cases like Back-to-Back Test and Regression Test. For generating tests which take the desired behavior of the system or software into account, we need to provide information about the requirements in a machine-readable language.
Author: Markus Gros / Wolfgang Meincke,
BTC Embedded Systems AG