*** Terms of Use ***
This artifact can be used for academic research purposes only. If you use it, you should cite the following FSE'18 paper, giving complete citation details as follows:
Rachel Tzoref-Brill and Shahar Maoz. "Modify, Enhance, Select: Co-Evolution of Combinatorial Models and Test Plans". In Proceedings of the 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’18), pp. 235-245, ACM, 2018.
*** Artifact Description ***
This collection contains 48 real-world industrial combinatorial models, with 116 versions and 68 version commits. 32 models have 2 versions, 13 models have 3 versions, 2 models have 4 versions, and one model has 5 versions. The models originate from IBM and were written by different combinatorial testing engineers over a period of 9 years, and originate from 13 different domains: firmware, PaaS, IaaS, file system, operating system, database, storage, analytics, banking, telecom, security, networking, and software applications (email, document management, finance, etc.). The models also capture different levels of testing, such as function test, system test, etc.
We did not select the models according to any criteria, and they represent all data available to us from IBM. The models were obfuscated in order not to disclose any confidential information.
The folder of the first version of each model contains its model file only. Any folder of a subsequent model version contains the combinatorial test plan of the previous model version, located in the subfolder "reportInputs". This test plan is given as input to the co-evolution alternatives generation. The test plan is in CSV format, and its file name is "pairwise.csv". Note that for most models, pairwise requirements were used, but there are a couple of models with more complex coverage requirements, as coded in the model file. In some cases, a second file called "pairwise_fixed.csv" resides next to "pairwise.csv". This is the same test plan from the previous version, only that the value combinations that were invalidated by changes in constraints made in the current model version are replaced with empty cells. "pairwise_fixed.csv" is given as input to the modify-enhance-select co-evolution alternative. When the model contains definitions of bad path combinations, "pairwise-badPath.csv" and "pairwise-badPath_fixed.csv" files may reside in the folder "reportInputs" as well, representing the bad path test plans of the previous model version.
*** Artifact Format ***
The model files are in XML format. Important tags are as follows:
- attribute: represents a parameter of the model
- value: represents a value of its enclosing attribute
- restriction: represents a constraint in the model
- coverage-reqs-set: a coverage requirement for the combinatorial test plan. It consists of a set of "coverage-req" tags.
- coverage-req: a component in its enclosing requirement coverage-reqs-set. Defines a subset of the parameters and an interaction level. This means that the resulting test plan must cover the required interaction level for this subset of parameters.
For example, the XML below defines a model specifying the following:
1. Three parameters of type string: A0, A1, and A2.
2. A0 and A1 have values V0 and V1. A2 has values V0, V1, and V2.
3. A constraint that excludes from the test space all tests where A1 equals A2
4. A pairwise coverage requirement on all parameters.
Additional optional tags that may be considered:
- negative-combination: defines a negative value combination (an error conditions). Such combinations will appear only in the bad path test plan, each in a separate test. For example, the following XML defines a negative combination of size 1: the value V1 of parameter A5.
- neg-coverage-reqs-sets: in all models in this collection, whenever negative combinations are defined, the associated coverage requirement for them is to cover every negative combination by a separate test (every 1 coverage). The following XML defines this coverage requirement.