The challenge

The University of Tasmania has a long-standing record with “big data” as the second oldest Data Migration Facility site in Australia which has allowed the university to store research data at scale since the 1990s. Over the past decades, the University of Tasmania has developed a chronic small file problem: a cluster of small files had started to accumulate into a “big” data problem. The goal was for Atos to introduce the High-Performance Storage System (HPSS) tier 1 Research Data Management System providing a global namespace as well as being proven to scale, allowing community collaboration with the source-code. Furthermore, there was a requirement to have direct access to HPSS experts and developers with long-standing history and integrity.

The solution

HPSS is the product of a 26-year collaboration consisting of IBM and five national laboratories of the U. S. Department of Energy. One of the great strengths of this collaboration is that the product developers are users, who focus on what is required and what works. The partnership is based upon the premise that no single organization has the ability to confront all the issues that must be resolved for significant advances in high-performance storage system technology. Currently, there are 28 sites having greater than ten petabytes, 13 greater than 50 petabytes, and five sites that exceed 100 petabytes, and one greater than 450 petabytes. Significantly, all the leading sites anticipate growth to several times current size without stressing the existing software, and the collaboration is actively working on the development of HPSS releases that will provide growth of decimal orders of magnitude over the current version.

Business benefit

By adapting Atos’ implementation of the HPSS solution, the University of Tasmania has ensured the use of three spatially separated sites. The HPSS solution has also enabled encrypted data-at-rest as well as global, scalable POSIX-style namespace for billions of directories and files. Furthermore, the implementation of HPSS by Atos afforded UTAS with scalable client endpoints for performance and resiliency as well as upload and download by design Another advantage was provision of the addition of checksums on file ingestion providing end-to-end data integrity. The collaboration between Atos and UTAS has been beneficial in being a directly accessible, proven to scale solution magnifying UTAS’s success from setback to comeback.