Cost-Efficient Sampling for Performance Prediction of Configurable Systems

A key challenge of the development and maintenance
of configurable systems is to predict the performance of
individual system variants based on the features selected. It is
usually infeasible to measure the performance of all possible variants,
due to feature combinatorics. Previous approaches predict
performance based on small samples of measured variants, but
it is still open how to dynamically determine an ideal sample
that balances prediction accuracy and measurement effort. In
this paper, we adapt two widely-used sampling strategies for
performance prediction to the domain of configurable systems
and evaluate them in terms of sampling cost, which considers
prediction accuracy and measurement effort simultaneously. To
generate an initial sample, we introduce a new heuristic based on
feature frequencies and compare it to a traditional method based
on t-way feature coverage. We conduct experiments on six real-world
systems and provide guidelines for stakeholders to predict
performance by sampling.