The HPSS Collaboration between IBM Houston Global Services and what are now five DOE National Laboratories (Lawrence Berkeley, Lawrence Livermore, Los Alamos, Oak Ridge, and Sandia) began in the fall of 1992. The goal was to produce a highly scalable high performance storage system. The High Performance Storage System (HPSS) needed to provide scalable hierarchical storage management (HSM), archive, and file system services. No product meeting the requirements existed. When HPSS design and implementation began scientific computing power and storage capabilities at a site, such as a DOE national laboratory, was measured in a few 10s of gigaops, data archived in HSMs in a few 10s of terabytes at most, data throughput rates to an HSM in a few megabytes/s, and daily throughput with the HSM in a few gigabytes/day. At that time, the DOE national laboratory and IBM HPSS design team recognized that we were headed for a data storage explosion driven by computing power rising to teraops/petaops requiring data stored in HSMs to rise to petabytes and beyond, data transfer rates with the HSM to rise to gigabytes/s and higher, and daily throughput with a HSM in 10s of terabytes/day. Therefore, we set out to design and deploy a system that would scale by a factor of 1,000 or more and evolve from the base above toward these expected targets and beyond.
Because of the highly scalable HPSS architecture these targets have been successfully met. We now recognize that computing power will rise to exaops by about 2020 with a corresponding rise in the need to scale storage in its various dimensions by another factor of 1,000. Further, other major application domains, such as real-time data collection, also require such extreme scale storage. We believe the HPSS architecture and basic implementation, built around a scalable relational database management system (IBM’s DB2) make it well suited to this challenge.
For a distributed collaboration such as the HPSS Collaboration that is producing a major software system to be successful careful thought went into its basic organization. Its basic governing document is a Collaboration Agreement spelling out intellectual property rights of the development partners and their management and organization. IBM has the responsibilities for commercialization and deployment, outside the development partners. There is an Executive Committee co-chaired by IBM and DOE lab representatives that sets major development and other policies. This group meets several times a year, primarily by teleconference. HPSS development is overseen by a Technical Committee coordinated by an IBM project manager. This group is generally organized around the major architectural modules of the system. It meets weekly, and more often as needed, by teleconference and once or twice a year in person. Development of the system follows industry standard software engineering practices and has an SEI CMM Level 3 rating. Following these software engineering practices has been a major factor in its success in producing a stable maintainable product.
The HPSS collaboration is based on the premise that no single organization has the experience and resources to meet all the challenges represented by the growing imbalance between computing power and data collection capabilities, and storage system I/O, capacity, and functionality. Over 20 organizations including industry, Department of Energy (DOE), other federal laboratories, universities, National Science Foundation (NSF) supercomputer centers and French Commissariat a l'Energie Atomique (CEA) have contributed to various aspects of this effort.
The primary HPSS development team consists of
- IBM Global Business Services (Houston, TX)
- Los Alamos National Laboratory (Los Alamos, NM)
- Lawrence Livermore National Laboratory (Livermore, CA)
- Lawrence Berkeley National Energy Research Supercomputer Center (Berkeley, CA)
- Oak Ridge National Laboratory (Oak Ridge, TN)
- Sandia National Laboratories (Albuquerque, NM)
- Commissariat a l'Energie Atomique, Direction des Applications Militaires (Bruyeres le Chatel, France)
- Gleicher Enterprises (Tucson, AZ)
|2015 HPSS Users Forum - The HPSS User Forum 2015 will be hosted by SciNet in Toronto, Canada from Monday, September 28 through Friday, October 2. For more information.|
|2015 HPSS Training - The next HPSS System Administration course from August 24th - 28th. For more information and registration.|
|2014 HPSS Users Forum - The HPSS User Forum 2014 will be hosted by Rechenzentrum Garching (RZG) in Munich, Germany from Monday, October 27 through Friday, October 31. For more information and registration.|
|HPSS at SC13 - SC13 is the 2013 international conference for high performance computing, networking, storage and analysis. SC13 will be in Denver, Colorado, from November 18th through 21st - Learn More. Come visit the HPSS folks at the IBM booth and schedule an HPSS briefing at the IBM Executive Briefing Center|
|2013 HPSS Users Forum - The 2013 HPSS Users Forum (HUF) will be hosted by the National Center for Atmospheric Research (NCAR), in Boulder, Colorado. The conference will run from November 4th - 7th. For more information and registration.|
|HPSS @ SEG13 - SEG13 is the 2013 annual meeting for the Society of Exploration Geophysics International Exposition. SEG13 will bring together all of the major companies involved in oil, gas, and mineral exploration, and new areas of civil engineering, environmental and archaeology. The conference will be held at the George R. Brown Convention Center in Houston, Texas, from September 22nd through 25th - Learn More. Come visit the HPSS folks at the IBM booth and schedule an HPSS briefing.|
|2013 HPSS Training - The next HPSS System Administration course from September 23rd - 27th. For more information and registration.|
|NCSA in production with RAIT - A massive 380 petabyte HPSS system was successfully deployed. -- the world's largest automated near-line data repository for open science. Learn more from NCSA, and HPCwire. The new HPSS system went into production using HPSS Redundant Array of Independent Tapes (RAIT) tiers, which is similar to RAID, providing redundancy for a tape stripe. RAIT allows HPSS customers to meet their performance and redundancy requirements without doubling their tape cost. Learn more about RAIT.|
|HPSS VFS for RHEL 6 is available - HPSS Virtual File System (VFS) allows a computer to use the UNIX mount command to mount the HPSS name space, much like NFS is mounted. Learn more about VFS.|