Evaluating the number of realizations (train-test splits) necessary to ensure robust estimates of conventional hydrothermal resource favorability using Monte Carlo cross-validation, the Great Basin, Western USA

Abstract

Supervised machine learning approaches used to produce favorability maps in resource assessments commonly partition an input dataset into training and testing subsets (that is, through train-test splits). However, if a single split is not representative of underlying resource conditions, the resulting favorability map may be biased, particularly with sparse or heterogeneous datasets. To mitigate the potential for unrepresentative favorability maps, recent works by U.S. Geological Survey researchers have used Monte Carlo cross-validation to implement a large number ( n) of randomized train-test splits to produce n realizations. The realizations are then aggregated using cell-wise median predictions to produce the resulting final favorability map, recognizing that the median will remove extreme values that might have been estimated from a comparatively few unrepresentative realizations. Theoretically, the median prediction at every cell converges to a single estimate as n approaches infinity, and the question addressed herein is: How many realizations are necessary to get a robust estimate (that is, an estimate unaffected by additional realizations)? Therefore, convergence metrics are described to identify the number of train-test splits and realizations needed to make a reliable map using the data and supervised machine learning workflow for conventional hydrothermal resource assessment of the Great Basin of the Western United States. For this study, robust prediction occurs with n approximately equal to 600 realizations, and smaller ensembles (for example, n is approximately equal to 200 or even 100) may be suitable during preliminary or exploratory analyses. Although the resulting threshold numbers will vary depending on the data and modeling approach used, the convergence metrics proposed and applied herein are applicable to other supervised machine learning studies that use Monte Carlo cross-validation.