“We needed a way to scale both the delivery of data and our development, because we can’t have a closed system that only a few developers can access and improve,” Martin said. 

To accomplish this goal, Prout and Leland used two readily available tools: Kubernetes, a containerized application software with native management and deployment capabilities, and the Themis storage system, which is in an open security storage enclave managed by the OLCF. This enabled them to create a pipeline that automatically builds and deploys APPL’s hyperspectral imaging application and provides secure, accessible and local data storage. The method is described in a technical report published earlier this year.

The project was an important pilot for both facilities. For APPL, the collaboration is a critical step in making its data more accessible to external users and paving the way for continued streamlining of its data management. 

“Our biggest success in all this was that we demonstrated the ability to use an open and scalable system for our data,” Martin said. The result gets to the heart of APPL’s mission to accelerate the transformation of plants as hardy feedstocks for bioenergy and bioproducts, for agriculture and for climate resilience. 

For the OLCF, which primarily offers HPC resources and support, working with APPL was an opportunity to develop, test and deploy scalable and secure data services for users outside the facility, expanding its support capabilities for open research. 

“There’s a lot of good science that could be enabled by these data services that otherwise might not be possible or would take much more time,” Leland said.

The collaboration is also a promising example of the enhanced capabilities at DOE facilities that will be possible through IRI.

“Pairing reliable, scalable and flexible data services with the OLCF’s computing resources is really unique,” Prout said. “I hope that this work ultimately contributes to the future of science and scientific discovery.”

The work was supported by the Laboratory Directed Research and Development Program at ORNL. 

UT-Battelle manages ORNL for DOE’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. The Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit energy.gov/science. – Betsy Sonewald

This Oak Ridge National Laboratory news article "Cross-facility collaboration grows new capabilities" was originally found on https://www.ornl.gov/news