High Performance Storage System

HPSS Logo
Incremental Scalability
Based on storage needs and deployment schedules, HPSS scales incrementally by adding computer, network and storage resources. A single HPSS namespace can scale from petabytes of data to exabytes of data, from millions of files to billions of files, and from a few file-creates per second to thousands of file-creates per second.
About HPSS   :   HPSS Collaboration

The HPSS Collaboration between IBM and what are now five DOE National Laboratories (Lawrence Berkeley, Lawrence Livermore, Los Alamos, Oak Ridge, and Sandia) began in the fall of 1992. The goal was to produce a highly scalable high performance storage system. The High Performance Storage System (HPSS) needed to provide scalable hierarchical storage management (HSM), archive, and file system services. No product meeting the requirements existed. When HPSS design and implementation began scientific computing power and storage capabilities at a site, such as a DOE national laboratory, was measured in a few tens of gigaflops, data archived in HSMs in a few tens of terabytes at most, data throughput rates to an HSM in a few megabytes/sec, and daily throughput with the HSM in a few gigabytes/day. At that time, the DOE national laboratory and IBM HPSS design team recognized that we were headed for a data storage explosion driven by computing power rising to teraflops/petaflops requiring data stored in HSMs to rise to petabytes and beyond, data transfer rates with the HSM to rise to gigabytes/sec and higher, and daily throughput with an HSM in tens of terabytes/day. Therefore, we set out to design and deploy a system that would scale by a factor of 1,000 or more and evolve from the base above toward these expected targets and beyond.

Because of the highly scalable HPSS architecture, these targets have been successfully met. We now recognize that computing power will rise to exaflops before 2022 with a corresponding rise in the need to scale storage in its various dimensions by another factor of 1,000. Further, other major application domains, such as real-time data collection, also require such extreme-scale storage. We believe the HPSS architecture and basic implementation, built around a scalable relational database management system (IBM’s Db2) make it well suited to this challenge.

For a distributed collaboration such as the HPSS Collaboration that is producing a major software system to be successful, careful thought went into its basic organization. Its basic governing document is a Collaboration Agreement spelling out intellectual property rights of the development partners and their management and organization. IBM has the responsibilities for commercialization and deployment, outside the development partners. There is an Executive Committee co-chaired by IBM and DOE lab representatives that sets major development and other policies. This group meets several times a year, primarily by teleconference. HPSS development is overseen by a Technical Committee coordinated by an IBM project manager. This group is generally organized around the major architectural modules of the system. It meets weekly, and more often as needed, by teleconference and once or twice a year in person. Development of the system follows industry standard software engineering practices. Following these software engineering practices has been a major factor in its success in producing a stable maintainable product.

The HPSS collaboration is based on the premise that no single organization has the experience and resources to meet all the challenges represented by the growing imbalance between computing power and data collection capabilities, and storage system I/O, capacity, and functionality. Over 20 organizations including industry, Department of Energy (DOE), other federal laboratories, universities, National Science Foundation (NSF) supercomputer centers, French Commissariat a l'Energie Atomique (CEA) and Gleicher Enterprises have contributed to various aspects of this effort. Today, the primary HPSS development team consists of:

  • IBM Consulting (Houston, TX)
  • Lawrence Berkeley National Energy Research Supercomputer Center (Berkeley, CA)
  • Lawrence Livermore National Laboratory (Livermore, CA)
  • Los Alamos National Laboratory (Los Alamos, NM)
  • Oak Ridge National Laboratory (Oak Ridge, TN)
  • Sandia National Laboratories (Albuquerque, NM)

< Home

Come meet with us!
HPSS @ ISC 2022
ISC 2022 is the event for high performance computing, machine learning, and data analytics, and will be in Frankfurt, Germany from May 29th through June 2nd, 2022 - Learn More. As we have done each year (pre-pandemic), we are scheduling and meeting with folks attending the conference. Please contact us to schedule a HPSS Single Client Briefing to meet with the IBM business and technical leaders of HPSS.

HPSS @ STS 2022 - postponed
The 4th Annual Storage Technology Showcase is in the planning phase, but HPSS expects to support the event later this year. Check out their web site - Learn More.

HPSS @ MSST 2022
The 37th International Conference on Massive Storage Systems and Technology will be in Santa Clara, California in May of 2022 - Learn More. Please contact us if you would like to meet with the IBM business and technical leaders of HPSS at Santa Clara University.

2022 HUF
The 2022 HPSS User Forum (HUF) is in the planning phase. The 2021 HUF was hosted online for six days spread across three weeks in October 2021 with no admission cost. We are planning to meet in person this year. Please check back next quarter for details. This will be a great opportunity to hear from HPSS users, collaboration developers, testers, support folks and leadership (from IBM and DOE Labs). Please contact us if you are not a customer but would like to attend.

HPSS @ SC22
The 2022 international conference for high performance computing, networking, storage and analysis will be in Dallas, TX from November 14th through 17th, 2022 - Learn More. As we have each year (pre-pandemic), we are scheduling and meeting with customers via IBM Single Client Briefings. Please contact your local IBM client executive or contact us to schedule a HPSS Single Client Briefing to meet with the IBM business and technical leaders of HPSS.

What's New?
HPSS 9.3 Release - HPSS 9.3 was released on December 14th, 2021 and introduces eight new features and numerous minor updates.

HUF 2021 - The HPSS User Forum was hosted virtually at no cost in October 2021.

DOE Announces HPSS Milestone - Todd Heer, Deputy Program Lead, Advanced Simulation and Computing (ASC) Facilities, Operations, and User Support (FOUS), announced that DOE High Performance Storage Systems (HPSS) eclipse one exabyte in stored data.

Atos Press Release - Atos boosts Météo-France’s data storage capacity to over 1 exabyte in 2025 to improve numerical modeling and climate predictions. Want to read more?

HPSS 9.2 Release - HPSS 9.2 was released on May 11th, 2021 and introduces eight new features and numerous minor updates.

Capacity Leader - ECMWF (European Center for Medium-Range Weather Forecasts) has a single HPSS namespace with over 650 PB spanning over 439 million files.

File-Count Leader - LLNL (Lawrence Livermore National Laboratory) has a single HPSS namespace with over 70 PB spanning 1,638 billion files.

Older News - Want to read more?
  • LLNL"
  • LANL"
  • NERSC"
  • ORNL"
  • Sandia"
  • IBM"
  • ANL"
  • Boeing"
  • BNL"
  • CEA"
  • CNES"
  • DWD"
  • DKRZ"
  • ECMWF"
  • PNNL
  • HLRS"
  • IU"
  • IITM"
  • IN2P3"
  • JAXA"
  • KEK"
  • KIT"
  • Met
  • MPCDF"
  • Meteo
  • NASA
  • NASA
  • NCMRWF"
  • NOAA
  • NOAA
  • NOAA
  • NOAA
  • Purdue"
  • SciNet"
  • SSC"
  • SLAC"
  • UTAS"
Home    |    About HPSS    |    Services    |    Contact us
Copyright 1992 - 2021, HPSS Collaboration. All Rights Reserved.