USGS Science⌕ Search
Home/Archive/USGS

USGS · 70189267

A refined index of model performance: a rejoinder

David R. Legates·Gregory J. McCabe
View original source ↗Export metadata ↓

Abstract

Willmott et al. [Willmott CJ, Robeson SM, Matsuura K. 2012. A refined index of model performance. International Journal of Climatology , forthcoming. DOI:10.1002/joc.2419.] recently suggest a refined index of model performance ( d r ) that they purport to be superior to other methods. Their refined index ranges from − 1.0 to 1.0 to resemble a correlation coefficient, but it is merely a linear rescaling of our modified coefficient of efficiency ( E 1 ) over the positive portion of the domain of d r . We disagree with Willmott et al. ( 2012 ) that d r provides a better interpretation; rather, E 1 is more easily interpreted such that a value of E 1 = 1.0 indicates a perfect model (no errors) while E 1 = 0.0 indicates a model that is no better than the baseline comparison (usually the observed mean). Negative values of E 1 (and, for that matter, d r < 0.5) indicate a substantially flawed model as they simply describe a ‘level of inefficacy’ for a model that is worse than the comparison baseline. Moreover, while d r is piecewise continuous, it is not continuous through the second and higher derivatives. We explain why the coefficient of efficiency ( E or E 2 ) and its modified form ( E 1 ) are superior and preferable to many other statistics, including d r , because of intuitive interpretability and because these indices have a fundamental meaning at zero. We also expand on the discussion begun by Garrick et al. [Garrick M, Cunnane C, Nash JE. 1978. A criterion of efficiency for rainfall-runoff models. Journal of Hydrology 36 : 375-381.] and continued by Legates and McCabe [Legates DR, McCabe GJ. 1999. Evaluating the use of “goodness-of-fit” measures in hydrologic and hydroclimatic model validation. Water Resources Research 35 (1): 233-241.] and Schaefli and Gupta [Schaefli B, Gupta HV. 2007. Do Nash values have value? Hydrological Processes 21 : 2075-2080. DOI: 10.1002/hyp.6825.]. This important discussion focuses on the appropriate baseline comparison to use, and why the observed mean often may be an inadequate choice for model evaluation and development.

Explore related subjects

International Journal of ClimatologyArticle

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

David R. Legates, Gregory J. McCabe. 2012-04-18. A refined index of model performance: a rejoinder. https://doi.org/10.1002/joc.3487

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related USGS reports

USGS2026

Tropicalization of the temperate zone: Spatiotemporal variability of winter warming and declining freeze days across the United States

We investigate changes in cool-season and winter daily minimum (T min ) and maximum (T max ) temperatures, and the occurrence of freeze days, from 1952 to 2024 across the conterminous United States (CONUS). Emphasis is placed on the tropical-temperate transition zone (TTTz) in the southeastern CONUS. During winter, ~70% of the land area exhibited T min warming rates exceeding those of T max . The countywide coldest T min became milder across 57% of the CONUS, while the coldest T max showed little change and even cooled east of the Rocky Mountains in the central CONUS. Across the TTTz, 75% of freeze days occur within a ~25–100-day window, often fewer than 75 days in the southernmost areas. Approximately 80% of counties exhibited significant contractions in freeze-day concentration, with the largest and most spatially consistent changes occurring in the Southeast, primarily driven by later start dates. Roughly 85% of the CONUS experienced a significant decline in freeze days, with the largest relative declines in regions where average winter T min is above freezing, while parts of the Pacific Northwest showed no significant change. An analysis of freeze day isopleths (30, 45, 60 and 75 days) across 20-year periods showed that the mean latitude of freeze days has migrated poleward substantially. Between 101° W and 79° W in the TTTz, the 30 freeze-day isopleth for the late period (2005–2024) was, on average, 122 km (~1.1° latitude) farther north than in the early period (1952–1971). Generally, the largest latitudinal shifts and percentage losses in freeze days occurred across low-elevation, low-relief regions at lower latitudes (e.g., the Mississippi River Valley), with abrupt shifts occurring near topographic gradients. Regions with sharp elevational gradients (e.g., Balcones Escarpment, Ouachita Mountains and Tennessee Valley) exhibited smaller temporal changes, likely reflecting the barrier-like influence of higher terrain on the poleward retreat of freeze days.

conterminous United States↗
USGS2022

A hydrologic perspective of major U.S. droughts

Drought is a recurring natural hazard that has substantial human and environmental impacts. Given continued global warming and associated climate change, there is concern that droughts could become more severe and longer lasting. To better monitor and understand drought development and persistence, it is helpful to understand the development and climatic drivers of past droughts. In this study we use monthly runoff percentiles to identify five major drought events in the conterminous United States (CONUS) from 1901 through 2020. For each drought event we examined spatial patterns of departures of mean monthly precipitation, temperature, soil moisture storage, and runoff for 2,107 hydrologic units (HUs) across the CONUS. Results indicated that precipitation deficits have been the primary driver of past major-drought events and temperature a secondary driver, even of the most recent drought event (September 1999 through September 2015) when positive temperature anomalies occurred over most of the CONUS. Additionally, negative soil moisture storage departures were more negative than runoff departures during the five drought events we examined, which emphasizes the importance of measuring both runoff and soil moisture to monitor drought conditions. We also examined the use of statistical persistence to develop short-term (i.e., 1 month) forecasts of runoff drought conditions in the CONUS by developing autoregressive integrated moving average (ARIMA) models for each HU. Results indicated that persistence can be used to predict short-term changes in the spatial pattern of drought and the areal extent of drought, but that predictions of runoff magnitude for any particular site are often poor.

International Journal of Climatology↗
USGS2021

Multi-year hydroclimatic droughts and pluvials across the conterminous United States

Time series of water‐year runoff for 2,109 hydrologic units (HUs) across the conterminous United States (CONUS) for the 1900 through 2014 period were used to identify drought and pluvial (i.e., wet) periods. Characteristics of the drought and pluvial events including frequency, duration, and severity were examined and compared. Additionally, a similar analysis was performed using gridded tree‐ring reconstructions of the Palmer Drought Severity Index (PDSI) for the period 1475 through 2005 to place the drought and pluvial characteristics determined using water‐year runoff for 1900 through 2014 in the context of multi‐century climate variability. The temporal and spatial variability of droughts and pluvials determined using runoff for the 1900 through 2014 period indicated that most drought events in the CONUS occurred before about 1970, whereas most pluvial periods occurred after about 1970. This change in the frequencies of drought and pluvial events around 1970 was largely related to an increase in fall (October through December) precipitation across much of the central United States. Also, the duration and severity of droughts and pluvials identified using runoff for the 1900 through 2014 period generally were not significantly different from the drought and pluvial characteristics identified using the PDSI for the 1475 through 2005 period.

International Journal of Climatology↗