Storage

Description

The NAS Filesystem on Ahalama is available as follow:

Location	Size	Description	Environment variable	Backup/Purge policy
/home	34TB (500GB/user, 3TB/group)	home directories	$HOME	None/None
/ home	TBD	group space ???	n/a	None/None
/		Contains one directory per group ???		None/None
/		contains one directory per user ???		None/None

Recommended usage

The following table gives examples of typical usage for the different filesystems. They are just recommendations, though. There's no functional difference between the mountpoints.

/home	for users' files and code/scripts, personal software installations or libraries
/*?/groups	for group-shared datasets, code and group software installations/libraries
/*?/users	for users jobs' input and output files

Backup / Purge Policies

The amount of storage available on Ahalama is way beyond our current backup infrastructure's capacity. As a consequence, we cannot provide any backup for files stored on the Ahalama filesystems. We insist on the fact that each user is responsible for his/her own backups, and that we won't be able to restore any file in case of accidental deletion and/or filesystem crash. We strongly recommend that each user organize and maintain his/her own backup policy, with scheduled transfers of important results or code from the cluster to other storage media.

For the same reasons, we do not purge nor delete anything from the cluster's filesystems. Each user is responsible for cleaning his/her own old and unused files, to avoid reaching the user or group's quota limits.

Quotas

Different quotas are enabled on the shared filesystems:

Filesystem	Total size	Blocks quota (soft/hard)	Inodes quota (soft/hard)	applies to	grace period
/home	32TB	500GB/700GB	?	user	30 days
/home	32TB	3TB/6TB	?	group	30 days

group quotas

Group quotas are shared between all the members of a group meaning that is userA and userB are both members of groupA, if userA uses 5TB on /home, userB won't be able to use more than 1TB. Note that if a user reaches his/her group quota, other group members won't be able to write new files until existing ones have been deleted.

block quotas

Blocks quota refers to the total amount of disk space consumed by all the files belonging to one user/group.

inodes quotas

Inodes quota refers to the total number files and directories owned by one user/group. One file consumes one inode.

hard limits

A hard limit specifies the maximum allocation for a user or group. For example, if you are allocated 10,000 files, then you cannot create any new files if you are over your quota. The only way to get under your quota is to delete files.

soft limits

Soft limits are used to provide a warning to users. If they go over their soft limit quota, then the grace time kicks in. The grace time says how long they have to remove files and get under their quota. If the grace time is set to seven days, then users have seven days to get under their quota. After the grace period, the soft limit is treated as a hard limit and the user is blocked from creating or updating new files.

How to check quotas

Checking quotas on NAS filesystem can be done using a simple wrapper script, which give a visual representation of the quotas usage:

$ quota
Filesystem   Quota     Usage
/home        50GB      24GB      ||||||||| 48%
/home      1024GB     341GB     |||||| 33%

Striping

Striping allows parts of files to be stored on different OSTs. A RAID 0 pattern, in which data is "striped" across a certain number of objects, is used; the number of objects is called the stripe_count. Each object contains "chunks" of data. When the "chunk" being written to a particular object exceeds the stripe_size, the next "chunk" of data in the file is stored on the next target. File striping presents several benefits. One is that the maximum file size is not limited by the size of a single target. NAS can stripe files over up to 160 targets, and each target can support a maximum disk use of 8 TB by a file. This leads to a maximum disk use of 1.48 PB by a file in NAS.

Another benefit of striped files is that the I/O bandwidth to a single file is the aggregate I/O bandwidth to the objects in a file and this can be as much as the bandwidth of up to 160 servers.

On the other hand, striping induces some overhead (both in terms of network and server latencies), and also a greater risk of loosing files in case of a fileystem failure: when you consider the example of striping each file across all servers, if one OSS crashes, a small part of every file it hosts is lost. By comparison, if each file has exactly one stripe, you lose fewer files, but you lose them in their entirety. Most users would rather lose some of their files entirely than all of their files partially.

Getting striping information

By default, files are not striped on Ahalama: each file is stored on a unique OST, but files from the same directory may be located on different servers.