Skip to content Skip to navigation

Oak Storage

Oak is a High Performance Computing (HPC) storage system available to research groups and projects at Stanford. Hosted by the Stanford Research Computing Center (SRCC) and sometimes referred as cheap and deep storage, Oak provides the research community with inexpensive storage for research projects, storage that can grow in order to accommodate the projects’ increasing storage requirements.

Please contact us at srcc-support@stanford.edu for any questions about Oak.

 

Oak Service Overview

Oak is available for use by all Stanford University and affiliated research projects and groups. This includes healthcare researchers, as well as researchers from the SLAC National Accelerator Laboratory (SLAC).

For more details about Oak, please look at the official Oak Service Description available here (Stanford only).

Two flavors of the service are available for purchase with a 4-year commitment:

Service flavor

Service description

Service price

Filespace

PIs/Faculty rent disk space in 10 TB increments.

$41.67 per 10TB / month

or $50 per TB / year

JBOD

PIs/Faculty purchase one or more full disk arrays (JBODs), with 350 TB usable capacity each, that is/are supported and administered by the SRCC team as part of the overall Oak service.

JBOD cost* + $666.67 per JBOD / month
(service only)

*JBOD pricing is ~$30k, dependent upon hard disk market variations (contact us for current pricing)

Oak storage is readily available from both the Sherlock and XStream HPC clusters, but also from multi-protocol gateways (Globus, SFTP, NFSv4, Samba/CIFS...). Restrictions may apply and personalized gateway service will incur additional costs to the PI.

Important! Oak is not suitable for data classified as Moderate or High Risk, and therefore is not a storage choice for any data that include PHI or PII.

 

Oak group space

  • There are only group/project spaces on Oak (similar to how /share/PI or /scratch/PI work on Sherlock).

  • When a group directory is created on Oak, we use the PI/faculty SUNet ID as the group name (eg. /oak/stanford/groups/SUNetID/).

  • Or, alternatively, a project directory can be created under /oak/stanford/projects/.

  • File system group quotas are used to limit the amount of file system space group/project can use.

  • Each group is provided with a personal dashboard that provides information about available disk space (quota) and usage.

 

Managing users

  • The PI provides an initial list of SUNet IDs having access to the new group or project shared directory on Oak.

  • The PI gets access to a Stanford Workgroup to manage authorized users (please be aware that changes made in Workgroup Manager may take up to 24 hours to be propagated to Oak).

  • Members of the workgroup can then organize files as they desire in the group/project directory (POSIX ACLs are supported). 

 

Accessing Oak from Sherlock and XStream

Oak storage is available from all nodes on Sherlock and XStream under /oak. Like Sherlock's /scratch, Oak is based on the Lustre parallel filesystem and is connected to Sherlock (1.0 and 2.0) and XStream through Infiniband.

Important! You need an account on both Oak and Sherlock (or XStream) to access Oak from Sherlock (or XStream). The environment variable $OAK should be defined on Sherlock and XStream and contains the path to your Oak group directory. You may also use the full path starting with /oak as described above.

 

Archiving files to Oak from Sherlock

The mpiFileUtils utilities are designed to copy files in parallel so you can quickly archives terabytes of data from scratch to Oak. The example below shows how to launch screen and launch a job that uses the dcp tool to copy a large directory:

[sunetid@sherlock-ln01 login_node ~]$ screen
[sunetid@sherlock-ln01 login_node ~]$ module load mpifileutils
[sunetid@sherlock-ln01 login_node ~]$ srun -p dev -n 2 dcp $SCRATCH/dir $OAK/scratch_archive/
 

If you're a Sherlock owner, you may want to replace `-p dev` with `-p your_partition` and increase the number of MPI tasks (`-n`) to copy even faster!

 

Can I access Oak from my desktop/laptop?

Yes, please see the Oak Gateways page (Stanford only).

 

Can I mount Oak on my own Linux cluster at Stanford?

Yes, the SRCC team can deploy specific Oak NFSv4 gateway(s) to mount your Oak group/project directory on your Linux-based compute cluster (or desktop for specific applications). It is mandatory to use SUNet IDs on your cluster and Kerberos is required to access Oak. Such a service will incur additional costs to the PI.

 

What about automatic backups?

While the hardware configuration is quite robust, Oak does not provide local or remote data backup, and should be considered as a single copy. The SRCC is currently evaluating options for adding automatic remote backups (to cloud storage). Such a service will incur additional costs to the PI.
 

Under the hood

Oak is a capacity-oriented HPC storage system designed for long term storage that doesn't rely on a single vendor implementation. It was designed by the SRCC team using COTS (commercial off-the-shelf) components and open source software to provide up to billions of inodes and tens of petabytes of storage space to answer Stanford researchers' big data storage needs.

The software of Oak is based on the Lustre filesystem and the Robinhood Policy Engine. Personalized usage dashboards are provided thanks to the Grafana and Docker projects.

Oak's scalable and highly available architecture is based on MD cells (metadata) and I/O cells (data) each with external SAS-3 switches and high-density JBOD chassis designed for the cloud market. The Lustre servers are interconnected through a high-bandwidth and low-latency Infiniband fabric (56 Gb/s links). Oak storage components are redundant, from servers to disk array controllers and data paths between servers and disks. It also provides disk mirroring for the metadata and double-parity RAID in a 8+2 configuration for the data (file content). The use of additional parity allows the storage system to continue to function even if two disks in a volume of ten disks fail simultaneously.

SRCC's Oak project was first presented at the Lustre Administrators and Developers workshop in September 2016 in Paris. The slides are available here.