Sean Cleveland, University of Hawai’i, May 15, 2018

A graduate student’s project studying ocean larval dispersion and population modeling would not run on the lab computers due to memory issues.  University of Hawaii ACI-REFs assisted in fixing the researcher’s code to utilize less memory, to run faster, and to increase the granularity of the time step to increase the level of accuracy in simulations results. The ACI-REFs further assisted the researcher by enabling parallel processing which reduced the time for computational runs from more than 870 days down to about 16 days – allowing the researcher to work at a scale and finer granularity that was previously untenable.  Prior to meeting with ACI-REFs the researchers had not thought of running their simulations on the UH High Performance Computing (HPC) cluster or realized the potential of parallelization in their implementation.

The Objective

In the summer of 2015 the Sean Cleveland and Ron Merrill (UH ACI-REFs) met with researcher Dr. Robert Toonen and his doctoral student, Johanna Wren,  from the Hawaiian Institute for Marine Biology (HIMB) to discuss how to assist with some code to run simulations for the first comprehensive estimate of connectivity of passive pelagic particles released from coral reef habitat throughout the Hawaiian Archipelago.. The simulations were using all the memory in their current lab workstations, but were not running to completion.  Since they had reached the capacity of their lab workstations for running the simulations, they met with ACI-REFs to see if they could utilize the recently deployed University of Hawaii High Performance Computing (HPC) cluster resource to help solve these problems. Johanna’s dissertation was dependent on completing the simulations so getting them to run and complete was the most important initial outcome for the researchers.

Once this initial problem was solved, the researchers were then interested in getting the simulation to run in a more reasonable time span, as it was taking weeks or months to complete.  And finally, a third goal was to increase the timestamp granularity for the simulation to get down to hourly time steps allow better resolution of movement and increasing the accuracy of the model.

The Solution

Sean Cleveland and Ron Merrill worked with Dr. Toonen and Johanna to gain access to the UH HPC cluster and for training on how to run their initial simulation code on the cluster. This allowed them to see how it worked on a compute node with resources larger than their entire existing lab compute infrastructure. This worked initially for the scope of simulations they were working on but as the scale for the length of the simulation increased they began to reach the capacity of the UH HPC compute node memory, which is 128GB.  David Schanzenbach, another UH ACI-REF, began working with Johanna to review the simulation code for ways to solve this memory issue. David discovered and fixed a memory leak which solved the issue, and also identified that the code could be parallelized to run faster.

Johanna and Dr. Toonen decided that they wanted to pursue the parallelization so that the simulation time step could become more granular, down to hourly steps allowing more accurate calculations of particle movement.  David helped get the simulation code to run parallel threads for portions of the simulation and was able to significantly reduce the time needed to run the simulation. With a threaded parallel version that could run on a UH HPC compute node, work that would have taken 870 days to run could now be completed in about 16 days.

The Result

As a result of efforts of the UH ACI-REF’s collaboration the researchers were able to: complete simulation execution without running out of memory, run the simulations in a shorter amount of time than initially would have been possible (16 vs 870 days) and finally the simulations could execute with a finer level of time step granularity resolving to better accuracy in model outputs.  Further, the enhanced computational model enabled Johanna, by May of 2016, to complete her Ph.D. dissertation work and publish the results of her work in PLoS ONE in December 2016.

The success of this collaboration paved the way for more researcher of HIMB to adopt using the UH HPC cluster and collaborate with the UH ACI-REFs to enhance their productivity.

Notable Publications and Presentations Resulting from this Work

Wren JLK, Kobayashi DR, Jia Y, Toonen RJ (2016) Modeled Population Connectivity across the Hawaiian Archipelago. PLoS ONE 11(12): e0167626. https://doi.org/10.1371/journal.pone.0167626

Collaborators and Resources

• Robert Toonen, Professor, SOEST, Hawaiian Institute for Marine Biology
• Johanna Wren, Ph.D, SOES, Hawaiian Institute for Marine Biology
• David Schanzenbach, ACI-REF, Cyberinfrastructure, UH Information Technology Services
• Ron Merrill,  ACI-REF, Cyberinfrastructure, UH Information Technology Services
• Sean Cleveland,  ACI-REF, Cyberinfrastructure, UH Information Technology Services

 

Resources:

• University of Hawaii High Performance Compute cluster.

Funding Sources

The work described in this case study was supported in part by a grant from the National Science Foundation, Award #1341935, Advanced Cyberinfrastructure – Research and Educational Facilitation: Campus-Based Computational Research Support and by NSF (OCE12-60169), and in part by a grant/cooperative agreement from the National Oceanic and Atmospheric Administration, Project R/SS-13, which is sponsored by the University of Hawaii Sea Grant College Program, SOEST, under Institutional Grant No. NA14OAR4170071 from NOAA Office of Sea Grant, Department of Commerce (for RJT and JLKW). Sea Grant contribution UNIHI-SEAGRANT-JC-13-16, SOEST contribution number 9638 and contribution number 1661 from the Hawaiʻi Institute of Marine Biology.