Simple, efficient allocation of modelling runs on heterogeneous clusters with MPI

David I. Donato

doi:10.1016/j.envsoft.2016.11.003

Home/Archive/USGS

USGS · 70178612

Simple, efficient allocation of modelling runs on heterogeneous clusters with MPI

David I. Donato

View original source ↗Export metadata ↓

Abstract

In scientific modelling and computation, the choice of an appropriate method for allocating tasks for parallel processing depends on the computational setting and on the nature of the computation. The allocation of independent but similar computational tasks, such as modelling runs or Monte Carlo trials, among the nodes of a heterogeneous computational cluster is a special case that has not been specifically evaluated previously. A simulation study shows that a method of on-demand (that is, worker-initiated) pulling from a bag of tasks in this case leads to reliably short makespans for computational jobs despite heterogeneity both within and between cluster nodes. A simple reference implementation in the C programming language with the Message Passing Interface (MPI) is provided.

Explore related subjects

Environmental Modelling and Software Article

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

David I. Donato. 2017. Simple, efficient allocation of modelling runs on heterogeneous clusters with MPI. https://doi.org/10.1016/j.envsoft.2016.11.003

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related USGS reports

USGS2026

A catalogue of Do's and Don'ts in the modeling of environmental systems

Modeling plays a vital role in understanding and managing complex environmental systems, but its credibility and quality depend heavily on a comprehensive set of defensible model activities and practices, especially when the system of interest is plagued with uncertainties and conflicting stakeholder perspectives. This paper proposes a catalogue of Do's and Don'ts to guide modelers in addressing the many pertinent considerations through the whole modeling cycle. This practical tool provides advice on approaching modeling effectively through adhering to good modeling practice. It emphasizes model choices that align with the model purpose and context, and the justification and documentation of modeling decisions and assumptions. Managing uncertainty is a core consideration. The identification, assessment and reporting of these uncertainties is important across the entire modeling process, which spans problem framing, technical design, implementation and application phases. Such good practices are critical for transparency and reliability of the modeling.

Environmental Modelling and Software↗

USGS2025

Automating physics-based models to estimate thermoelectric-power water use

Thermoelectric (TE) power plants withdraw more water than any other sector of water use in the United States and consume water at rates that can be significant especially in water-stressed regions. Historical TE water-use data have been inconsistent, incomplete, or discrepant, resulting in an increased research focus on improving the accuracy and availability of TE water-use data using modeling approaches. This paper describes and benchmarks new code that was developed to automate and update a physics-based TE water use model that was previously published. Utilizing the automated physics-based model, monthly TE-power water withdrawal and consumption were calculated for a total of 1341 TE power plants for the 2008–2020 historical reanalysis. The updated and automated physics-based thermoelectric-power water-use model provides spatially and temporally relevant TE water-use estimates that are consistent, reproducible, transparent, and can be generated efficiently for water-using, utility-scale TE-power plants across conterminous United States (CONUS).

Environmental Modelling and Software↗

USGS2024

Transfer learning with convolutional neural networks for hydrological streamline delineation

Hydrological streamline delineation is critical for effective environmental management, influencing agriculture sustainability, river dynamics, watershed planning, and more. This study develops a novel approach to combining transfer learning with convolutional neural networks that capitalize on image-based pre-trained models to improve the accuracy and transferability of streamline delineation. We evaluate the performance of eleven image-based pre-trained models and a baseline model using datasets from Rowan County, North Carolina, and Covington River, Virginia in the USA. Our results demonstrate that when models are adapted to a new area, the fine-tuned ImageNet pre-trained model exhibits superior predictive accuracy, markedly higher than the models trained from scratch or those only fine-tuned on the same area. Moreover, the pre-trained model achieves better smoothness and connectivity between classified streamline channels. These findings underline the effectiveness of transfer learning in enhancing the delineation of hydrological streamlines across varied geographies, offering a scalable solution for accurate and efficient environmental modelling.

North Carolina, Virginia↗