Comprehension and evolution of combinatorial test designs
Rachel Tzoref-Brill and Shahar Maoz
Research project since 2014, joint study of IBM Research Haifa and Tel Aviv University
Publications
ATVA'15: Lattice-Based Semantics for Combinatorial Model Evolution
ASE'16: Visualization of Combinatorial Models and Test Plans
ICSE'17: Syntactic and Semantic Differencing for Combinatorial Models of Test Designs
FSE'18: Modify, Enhance, Select: Co-Evolution of Combinatorial Models and Test Plans
Lattice-Based Semantics for Combinatorial Model Evolution
Combinatorial test design (CTD) is an effective test design technique, considered to be a testing best practice. CTD provides automatic test plan generation, but it requires a manual definition of the test space in the form of a combinatorial model. As the system under test evolves, e.g., due to iterative development processes and bug fixing, so does the test space, and thus, in the context of CTD, evolution translates into frequent manual model definition updates.
In this work, we show that the Boolean semantics currently in use by CTD tools to interpret the model is inadequate for combinatorial model evolution, and propose to replace it with a new lattice-based semantics that (1) provides a consistent interpretation of atomic changes to the model via Galois connections, and (2) exposes which additional parts of the model must change following an atomic change, in order to restore validity. We further use the new lattice-based semantics to define new higher-level atomic operations for combinatorial model evolution. Finally, we identify recurring abstraction and refinement patterns in the evolution of 42 real-world industrial models, and use the new lattice-based semantics to define new higher-level atomic constructs that encapsulate these patterns.
The proposed lattice-based semantics and related new modeling constructs advance the state-of-the-art in CTD with a new foundation for model evolution and with better tools for change comprehension and management.
R. Tzoref-Brill and S. Maoz , Lattice-Based Semantics for Combinatorial Model Evolution. Proc. of ATVA 2015.
Visualization of Combinatorial Models and Test Plans
Combinatorial test design (CTD) is an effective and widely used test design technique. CTD provides automatic test plan generation, but it requires a manual definition of the test space in the form of a combinatorial model. One challenge for successful application of CTD in practice relates to this manual model definition and maintenance process. Another challenge relates to the comprehension and use of the test plan generated by CTD for prioritization purposes.
In this work we introduce the use of visualizations as a means to address these challenges. We apply three different forms of visualization, matrices, graphs, and treemaps, to visualize the relationships between the different elements of the model, and to visualize the strength of each test in the test plan and the relationships between the different tests in terms of combinatorial coverage. We evaluate our visualizations via a user survey with 19 CTD practitioners, as well as via two industrial projects in which our visualization was used and allowed test designers to get vital insight into their models and into the coverage provided through CTD generated test plans.
R. Tzoref-Brill, Paul Wojciak, and S. Maoz , Visualization of Combinatorial Models and Test Plans. Proc. of ASE 2016.
Syntactic and Semantic Differencing
for
Combinatorial Models of Test Designs
Combinatorial test design (CTD) is an effective test design technique, considered to be a testing best practice. CTD provides automatic test plan generation, but it requires a manual definition of the test space in the form of a combinatorial model. As the system under test evolves, e.g., due to iterative development processes and bug fixing, so does the test space, and thus, in the context of CTD, evolution translates into frequent manual model definition updates. Manually reasoning about the differences between versions of real-world models following such updates is infeasible due to their complexity and size. Moreover, representing the differences is challenging.
In this work, we propose a first syntactic and semantic differencing technique for combinatorial models of test designs. We define a concise and canonical representation for differences between two models, and suggest a scalable algorithm for automatically computing and presenting it. We use our differencing technique to analyze the evolution of 42 real-world industrial models, demonstrating its applicability and scalability. Further, a user study with 16 CTD practitioners shows that comprehension of differences between real-world combinatorial model versions is challenging and that our differencing tool significantly improves the performance of less experienced practitioners. The analysis and user study provide evidence for the potential usefulness of our differencing approach.
Our work advances the state-of-the-art in CTD with better capabilities for change comprehension and management.
R. Tzoref-Brill and S. Maoz , Syntactic and Semantic Differencing for Combinatorial Models of Test Designs. Proc. of ICSE 2017.
Supporting materials
Modify, Enhance, Select:
Co-Evolution of Combinatorial Models and Test Plans
The evolution of software introduces many challenges to its testing. Considerable test maintenance efforts are dedicated to the adaptation of the tests to the changing software. As a result, over time, the test repository may inflate and drift away from an optimal test plan for the software version at hand. Combinatorial Testing (CT) is a well-known test design technique to achieve a small and effective test plan. It requires a manual definition of the test space in the form of a combinatorial model, and then automatically generates a test plan design, which maximizes the added value of each of the tests. CT is considered a best practice, however its applicability to evolving software is hardly explored.
In this work, we introduce a first co-evolution approach for combinatorial models and test plans. By combining three building blocks, to minimally modify existing tests, to enhance them, and to select from them, we provide five alternatives for co-evolving the test plan with the combinatorial model, considering tradeoffs between maximizing fine-grained reuse and minimizing total test plan size, all while meeting the required combinatorial coverage. We use our solution to co-evolve test plans of 48 real-world industrial models with 68 version commits. The results demonstrate the need for co-evolution as well as the efficiency and effectiveness of our approach and its implementation. We further report on an industrial project that found our co-evolution solution necessary to enable adoption of CT with an agile development process.
R. Tzoref-Brill and S. Maoz , Modify, Enhance, Select: Co-Evolution of Combinatorial Models and Test Plans. Proc. of FSE 2018.