Filesystems

Usage of the filesystems is subject to the Disk Usage Policy.

`/home`

Shared resource by all users of Boreal.
313 TB system.
Provides storage for source code, submission scripts, small input files and possibly executables.

Each project has a /dtemp/<project_id> directory for storing project-specific files.
Global 2.8 PB Lustre file system visible to all the compute nodes and the login nodes.
Users should direct program output to /dtemp and not /home. /dtemp is the preferred file system to store input, output and executables.
/dtemp has more space, higher bandwidth and lower latency than the /home file system.
/dtemp is for temporary storage it is not backed up. Files older than 60 days in the /dtemp file system are subject to deletion without notification.

/archive is a 22PB disk/tape library shared by several PNNL systems.
Boreal users have a /archive/<login_id> directory for storing input and output files and submission scripts.
The way users store files can influence retrieval time (on the order of weeks compared to days).
EFFICIENT STORAGE: small numbers of large files is the most efficient way of storing files on /archive.
INEFFICIENT STORAGE: large numbers of small files is the least efficient method. It can takes weeks to retrieve these files.
To store a large number of files, first tar czf <name>.tar.gz <dir> the files, then store the large file archive.
/archive is intended for output files you want to save and large input files you intend to reuse.

For more information on /archive, aka Aurora, see Data File Storage (Aurora).

Each Boreal node has a 246 GB local disk mounted as /scratch.
/scratch is used to store intermediate results, input files, and executables to reduce access time.
The /scratch file system has the lowest latency, highest per node bandwidth of all the file systems.
/scratch is volatile memory:
- files on /scratch will be deleted after the job terminates;
- to preserve files, move them from /scratch to /dtemp in the submission script (do not store them on /home);
- do not store files you want to save (e.g. checkpoint files) on /scratch because if the job terminates before the copy completes, the data will be lost.
Please direct output files to the /dtemp file system.

Each Boreal login and data mover node has the /emslfs filesystem mounted.
/emslfs is a shared filesystem that is pervasively mounted on EMSL systems. It is a convenient way to move data between systems. It should never be used for input or output of jobs.
/emslfs is mounted as a convenience for users who already have data there. Directories are not created for you automatically. Contact MSC Consulting if you need directories created.