Skip to Main content Skip to Navigation

Investigate the matrix : leveraging variability to specialize software and test suites

Paul Temple 1, 2 
Abstract : Nowadays, software have to be efficient, fast to execute, etc. They can be configured to adapt to specific needs. Each configuration leads to a different system and usually it is hard to generate them all. Thus, the exhaustive evaluation of their performance is impossible. Furthermore, several executions of systems, under different conditions, are needed to properly evaluate performances. Two dimensions emerge from this description of performance testing: the selection of system configurations allowing to generate associated systems that meet expressed needs and the selection of test cases allowing to observe performances of systems under different conditions. We propose to represent those two dimensions as a (performance) matrix: one dimension represents selected systems that can be observed while the other dimension represents the set of test cases that will be executed on each of these systems. Each cell is the execution of a program variant regarding a test. The contributions of this thesis are as follows : First, we leverage Machine Learning techniques in order to specialize a Software Product Line (in this case a video generator) helping in selecting a configuration that is likely to meet requirements. End users must be able to express their requirements such that it results in a binary decision problem (i.e., configurations that are acceptable and those that are not). Machine Learning techniques are then used to retrieve partial configurations that specialize a Software Product Line to guide end users and reduce the configuration space. In the end, this work aims at diminishing the first dimension of the matrix that deals with systems and programs. Second, we propose a new method assessing the ability of test suites to reveal significant performance differences of a set of configurations tackling the same task. This method can be used to assess whether a new test case is worth adding to a test suite or to select an optimal test set with respect to a property of interest. In the end, it might help structuring the execution of tests. For instance, it can create an order of execution resulting in using less test cases that are presented in the second dimension of the matrix. We evaluated our approach on several systems from different domains such as OpenCV or Haxe.
Keywords : Software Tests Matrix
Document type :
Complete list of metadata

Cited literature [141 references]  Display  Hide  Download
Contributor : ABES STAR :  Contact
Submitted on : Friday, May 10, 2019 - 5:14:06 PM
Last modification on : Friday, August 5, 2022 - 2:54:52 PM


Version validated by the jury (STAR)


  • HAL Id : tel-02125962, version 1


Paul Temple. Investigate the matrix : leveraging variability to specialize software and test suites. Software Engineering [cs.SE]. Université Rennes 1, 2018. English. ⟨NNT : 2018REN1S087⟩. ⟨tel-02125962⟩



Record views


Files downloads