Cesar Sul, University of Southern California, August 30, 2018

ACI-REFs at the University of Southern California’s Center for High-Performance Computing (HPC) worked with the USC-based Los Angeles Behavioral Economics Laboratory’s research team to move their fMRI workflow from their lab to HPC, and to train their graduate students during a period of staff transition.

The Objective

The USC-based Los Angeles Behavioral Economics Laboratory (LABEL), led by Isabelle Brocas and Juan Carrillo, does research on how decision-making capabilities change as people get older. Their goal is to understand why the ability to make effective decisions tends to decline as adults age and whether this is due to lifestyle conditions or changes in the brain.

To this end, LABEL collects functional MRI (fMRI) scans on subjects and uses behavioral neuroeconomics to fit them into a model to understand their preferences. A ten minute experiment on one participant results in 300 images. The project plans to collect the scans of 60 participants which will result in at least 18000 fMRI images that will have to be processed. Furthermore, the images are duplicated several times during various stages of processing with software tools like FSL (FMRIB’s1 Software Library) a set of domain specific software tools to analyze fMRI brain imaging data . The data were being stored on a local storage array and then copied to and processed on local desktop computers in the LABEL lab, resulting in the local computers being unavailable for long periods of time. In order to run even a small-scale analysis, researchers need to exert significant logistical effort to shift data back and forth between limited capacity short-term and long-term storage locations.

Based on a recommendation from their research assistant James Melrose, a Ph.D. student in psychology and a previous user of HPC’s compute cluster, LABEL decided to transition its workflow to HPC. Melrose was preparing to graduate soon and around this time research assistant Jekaterina Zyuzi, a Ph.D. student in neurology, had just been hired to take over the research. Melrose then faced two big tasks: one, to move the LABEL workflow from their local lab to HPC and, two, to ensure a smooth transition between himself and successor Zyuzin. Zyuzin asked ACI-REFs Cesar Sul and Erin Shaw if they could work with her during this stage of transition.

By moving their workflow to USC’s Center for High-Performance Computing (HPC) research computing environment, LABEL would gain the ability to store large datasets in a single location, process it in less time, and free up their lab and staff resources. The task was time-critical and required getting Zyuzin on-boarded while Melrose still had time to assist.

The Solution

ACI-REFs Cesar and Erin had worked with Melrose for two years, starting when he was a new research assistant on a clinical diabetes project with no research computing experience. Cesar and Erin had given Melrose and three colleagues training on how to use Linux and HPC resources to transfer data and run compute jobs through consultations, weekly office hours, help tickets, and one-on-one meetings. They helped his team install and run FSL and later, how to use Matlab and Python on HPC. While working to create a suitable HPC workflow, Melrose volunteered to collaborate with ACI-REFs to improve a workshop using Pegasus’ workflow management software. He provided the workflow, image data, and FSL programs, and assisted ACI-REFs in creating realistic examples that helped make the workshop more realistic and easier to understand.

Because of his previous experience with research computing, LABEL investigators asked Melrose to help them improve the LABEL workflow. Since this was the second time Melrose was tasked with moving a scientific workflow to HPC resources, he knew the right questions to ask ACI-REF and administrators, and was able to complete the transition faster this time. Melrose’s first goal was to install all the necessary packages for R, Python, and Matlab in his allocated project space using the applications available in the HPC maintained software repository. Using traditional desktop based resources it would take over a month to process 60 subjects, each with 5 scans. With these software tools running on HPC resources, LABEL was able to scale up from running analysis on a single subject at a time to running many subjects in parallel, cutting total processing time to around 12 hours.

In September 2017, as Melrose was wrapping up his Ph.D. research, graduate student Jekaterina Zyuzin joined LABEL. Zyuzin’s previous research experience was in a different domain, so her main priority was getting up to speed on her new project. Immediately she encountered issues accessing data within the shared project directory. Having been the sole member of the project, it was not necessary for Melrose to pay attention to file and file system permissions, so he needed to assist Zyuzin when issues came up. As his thesis defense date approached and with less time to address new issues, the ACI-REFs stepped in to help. After receiving consent from both Melrose and PI Brocas, HPC system administrator Chris Ho addressed the data access restriction, enabling access to the entire research group. ACI-REFs also provided training that would allow Melrose and Zyuzin to share files without needing further intervention from HPC staff. Future members of LABEL would also benefit and would only need to make a minor one-time environment setting to maintain shared permissions.

Besides simply accessing files, Zyuzin needed help getting acquainted with the remote HPC environment. She was used to desktop based research computing but had no previous experience with advanced cyber-infrastructure (ACI). ACI-REFS provided training through office hours and workshops to help Zyuzin learn to use a command line interface and submit batch jobs. This enabled her to focus on learning the new field instead of spending time researching how to use Unix.

The Result

ACI-REFs helped LABEL transition their workflow, share group files, and get new personnel working on the project. All of the problems encountered by the LABEL group could have been resolved by anyone sufficiently motivated given enough time, but by leveraging ACI-REFs as a resource, LABEL could quickly resolve their technical issues. This allowed their researchers to work together to enable a smooth transition of responsibilities and continuity of work. Both Jekaterina Zyuzin and James Melrose were then able to focus their efforts on their research problems and Ph.D. studies.

Zyuzin now manages the LABEL workflow on HPC resources. Using the data processed on HPC resources LABEL produced a poster that deals with identifying regions of the brain that help with complex decision making. There are plans to do a more sophisticated analysis along with a draft of a new paper.

Notable Publications and Presentations Resulting from this Work

Poster: “Determining Neural Correlates Responsible for Complex Value-Based Decision-Making” (by Jekaterina Zyuzin, James Melrose, T. Dalton Combs, Niree Kodaverdian, Max Ibrahimzade, Calvin Leather, John Monterosso, Mara Mather, Juan D. Carrillo and Isabelle Brocas)

Thesis: Homeostatic imbalance and monetary delay discounting: effects of feeding on RT, choice, and brain response Melrose, Andrew James

http://digitallibrary.usc.edu/cdm/compoundobject/collection/p15799coll40/id/490367/rec/2

USC HPC Researchers: Isabelle Brocas, Juan Carrillo,  Jekaterina Zyuzin, and James Melrose

USC ACI-REFs: Research Computing Facilitators Erin Shaw and Cesar Sul

Collaborators and Resources

• Isabella Brocas, Ph.D., Professor of Economics, University of Southern California, Los Angeles Behavioral Economics Laboratory
• Juan Carrillo, Ph.D., Professor of Economics, University of Southern California, Los Angeles Behavioral Economics Laboratory
• Jekaterina Zyuzin Graduate Student, University of Southern California
• James Melrose, Ph.D., University of Southern California
• Cesar Sul, Research computing facilitator, University of Southern California, Center for High-Performance Computing
• Erin Shaw, Research computing facilitator, University of Southern California, Center for High-Performance Computing
• Chris Ho, Systems Administrator, University of Southern California, Center for High-Performance Computing

Funding Sources

The work described in this case study was supported in part by a grant from the National Science Foundation, Award #1341935, Advanced Cyberinfrastructure – Research and Educational Facilitation: Campus-Based Computational Research Support and in part by USC’s Information Technology Services and Center for High-Performance Computing.

The HPC research project described was supported in part by the National Institute on Aging, Award #1R21AG046917-01A1, ”A neuroeconomic study of choice consistency in aging” (I. Brocas, J. Monterosso, M. Mather, J. Carrillo), 2015-2018. Institute for Economic Policy Research (IEPR), Economics Department