Supercomputing

Neural Network Training at Scale With Deep Learning Frameworks - From NASA@SC21

Improvements in instrument technology have resulted in enormous amounts of scientific data being generated on a regular basis. Currently, terabytes of data are produced from just a day’s worth of experiments or observations. This situation has created new opportunities to learn more about the universe and to think differently about data so that we can do better science, and NASA’s scientific community is becoming increasingly aware of modern industrial practices to leverage large amounts of data to answer key questions. With this fresh interest in data science, the High-End Computing Capability (HECC) Project’s Data Science team created a framework to facilitate the translation and formatting of data pipelines to work with the deep learning frameworks currently being developed in industry.

In deep learning, a subset of data science, nested neural networks are built to create models based on large datasets. Deep learning is an enabling technology for artificial intelligence and is credited for recent advances in areas such as image recognition, pattern recognition, and anomaly detection—tools that can assist NASA in the development of new technologies.

Project Details

The Data Science team aims to provide NASA researchers with deep learning tools to support advancements in scientific discovery by maintaining software stacks for use in the HECC environment at the NASA Advanced Supercomputing (NAS) facility. This environment supports software languages such as Python and R, and the software stacks are built using Anaconda, an open-source and extensible framework for managing various software packages and environments. We currently support eight different conda environments as part as the deep learning framework we maintain for our users. The framework caters to users’ varying requirements, providing them with ready access to environments that contain the latest versions of TensorFlow, PyTorch, and other popular utilities used in deep learning. Recently, we added the distributed training utility Horovod as a standalone environment that allows users to distribute their training across multiple nodes up to three times faster than using the TensorFlow training distribution method. This software stack is used on 49 NVIDIA V100 GPU nodes at the NAS facility, with a total of 200 GPUs.