A Novel Approach to Address External Validity Issues in Fault Prediction Using Bandit Algorithms

Teruki HAYAKAWA; Masateru TSUNODA; Koji TODA; Keitaro NAKASAI; Amjed TAHIR; Kwabena Ebo BENNIN; Akito MONDEN; Kenichi MATSUMOTO

doi:10.1587/transinf.2020EDL8098

IEICE TRANSACTIONS on Information

A Novel Approach to Address External Validity Issues in Fault Prediction Using Bandit Algorithms

Teruki HAYAKAWA, Masateru TSUNODA, Koji TODA, Keitaro NAKASAI, Amjed TAHIR, Kwabena Ebo BENNIN, Akito MONDEN, Kenichi MATSUMOTO

Full Text Views

0

Cite this

Summary :

Various software fault prediction models have been proposed in the past twenty years. Many studies have compared and evaluated existing prediction approaches in order to identify the most effective ones. However, in most cases, such models and techniques provide varying results, and their outcomes do not result in best possible performance across different datasets. This is mainly due to the diverse nature of software development projects, and therefore, there is a risk that the selected models lead to inconsistent results across multiple datasets. In this work, we propose the use of bandit algorithms in cases where the accuracy of the models are inconsistent across multiple datasets. In the experiment discussed in this work, we used four conventional prediction models, tested on three different dataset, and then selected the best possible model dynamically by applying bandit algorithms. We then compared our results with those obtained using majority voting. As a result, Epsilon-greedy with ϵ=0.3 showed the best or second-best prediction performance compared with using only one prediction model and majority voting. Our results showed that bandit algorithms can provide promising outcomes when used in fault prediction.

Publication: IEICE TRANSACTIONS on Information Vol.E104-D No.2 pp.327-331

Publication Date: 2021/02/01

Publicized: 2020/10/30

Online ISSN: 1745-1361

DOI: 10.1587/transinf.2020EDL8098

Type of Manuscript: LETTER

Category: Software Engineering

Authors

Teruki HAYAKAWA
  Kindai University
Masateru TSUNODA
  Kindai University
Koji TODA
  Fukuoka Institute of Technology
Keitaro NAKASAI
  Nara Institute of Science and Technology
Amjed TAHIR
  Massey University
Kwabena Ebo BENNIN
  Wageningen University & Research
Akito MONDEN
  Okayama University
Kenichi MATSUMOTO
  Nara Institute of Science and Technology

Keyword

defect prediction, multi-armed bandit, diversity of datasets, dynamic model selection, risk-based testing

Cite this

Copy

Teruki HAYAKAWA, Masateru TSUNODA, Koji TODA, Keitaro NAKASAI, Amjed TAHIR, Kwabena Ebo BENNIN, Akito MONDEN, Kenichi MATSUMOTO, "A Novel Approach to Address External Validity Issues in Fault Prediction Using Bandit Algorithms" in IEICE TRANSACTIONS on Information, vol. E104-D, no. 2, pp. 327-331, February 2021, doi: 10.1587/transinf.2020EDL8098.
Abstract: Various software fault prediction models have been proposed in the past twenty years. Many studies have compared and evaluated existing prediction approaches in order to identify the most effective ones. However, in most cases, such models and techniques provide varying results, and their outcomes do not result in best possible performance across different datasets. This is mainly due to the diverse nature of software development projects, and therefore, there is a risk that the selected models lead to inconsistent results across multiple datasets. In this work, we propose the use of bandit algorithms in cases where the accuracy of the models are inconsistent across multiple datasets. In the experiment discussed in this work, we used four conventional prediction models, tested on three different dataset, and then selected the best possible model dynamically by applying bandit algorithms. We then compared our results with those obtained using majority voting. As a result, Epsilon-greedy with ϵ=0.3 showed the best or second-best prediction performance compared with using only one prediction model and majority voting. Our results showed that bandit algorithms can provide promising outcomes when used in fault prediction.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.2020EDL8098/_p

Copy

@ARTICLE{e104-d_2_327,
author={Teruki HAYAKAWA, Masateru TSUNODA, Koji TODA, Keitaro NAKASAI, Amjed TAHIR, Kwabena Ebo BENNIN, Akito MONDEN, Kenichi MATSUMOTO, },
journal={IEICE TRANSACTIONS on Information},
title={A Novel Approach to Address External Validity Issues in Fault Prediction Using Bandit Algorithms},
year={2021},
volume={E104-D},
number={2},
pages={327-331},
abstract={Various software fault prediction models have been proposed in the past twenty years. Many studies have compared and evaluated existing prediction approaches in order to identify the most effective ones. However, in most cases, such models and techniques provide varying results, and their outcomes do not result in best possible performance across different datasets. This is mainly due to the diverse nature of software development projects, and therefore, there is a risk that the selected models lead to inconsistent results across multiple datasets. In this work, we propose the use of bandit algorithms in cases where the accuracy of the models are inconsistent across multiple datasets. In the experiment discussed in this work, we used four conventional prediction models, tested on three different dataset, and then selected the best possible model dynamically by applying bandit algorithms. We then compared our results with those obtained using majority voting. As a result, Epsilon-greedy with ϵ=0.3 showed the best or second-best prediction performance compared with using only one prediction model and majority voting. Our results showed that bandit algorithms can provide promising outcomes when used in fault prediction.},
keywords={},
doi={10.1587/transinf.2020EDL8098},
ISSN={1745-1361},
month={February},}

Copy

TY - JOUR
TI - A Novel Approach to Address External Validity Issues in Fault Prediction Using Bandit Algorithms
T2 - IEICE TRANSACTIONS on Information
SP - 327
EP - 331
AU - Teruki HAYAKAWA
AU - Masateru TSUNODA
AU - Koji TODA
AU - Keitaro NAKASAI
AU - Amjed TAHIR
AU - Kwabena Ebo BENNIN
AU - Akito MONDEN
AU - Kenichi MATSUMOTO
PY - 2021
DO - 10.1587/transinf.2020EDL8098
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E104-D
IS - 2
JA - IEICE TRANSACTIONS on Information
Y1 - February 2021
AB - Various software fault prediction models have been proposed in the past twenty years. Many studies have compared and evaluated existing prediction approaches in order to identify the most effective ones. However, in most cases, such models and techniques provide varying results, and their outcomes do not result in best possible performance across different datasets. This is mainly due to the diverse nature of software development projects, and therefore, there is a risk that the selected models lead to inconsistent results across multiple datasets. In this work, we propose the use of bandit algorithms in cases where the accuracy of the models are inconsistent across multiple datasets. In the experiment discussed in this work, we used four conventional prediction models, tested on three different dataset, and then selected the best possible model dynamically by applying bandit algorithms. We then compared our results with those obtained using majority voting. As a result, Epsilon-greedy with ϵ=0.3 showed the best or second-best prediction performance compared with using only one prediction model and majority voting. Our results showed that bandit algorithms can provide promising outcomes when used in fault prediction.
ER -

IEICE TRANSACTIONS on Information