Estimator for generalization performance of machine learning model trained by biased data collected from multiple references

Abstract

AbstractThe data acquired in civil engineering tasks often involve high acquisition costs, and the available datasets tend to have a limited number of samples and are highly biased. To estimate the performance of machine learning models, k‐fold cross‐validation (k‐CV) is widely used. However, if only limited data are available and the data distribution is biased, k‐CV tends to overestimate the performance for practical applications. This study proposed a new estimator, leave one reference out and k‐CV (LORO‐k‐CV), to determine the practical performance of machine learning models, that is, the generalization performance for population data in the target task, in case data are collected by multiple references resulting in biased data. LORO‐k‐CV is a combination of a new concept, LORO‐CV, that estimates the performance in the extrapolation region of the training data without human intervention and k‐CV, considering the ratio of the interpolation and extrapolation regions. The efficacy of LORO‐k‐CV was validated with its application to the regression task for the chloride‐ion concentration of concrete structures. To more specifically demonstrate the advantages of LORO‐k‐CV in model construction, the feature selections were conducted using both k‐CV and LORO‐k‐CV methods. These results revealed that LORO‐k‐CV can effectively construct a model with improved generalization performance even from the same data in cases where data are collected by multiple references, resulting in biased data.