Parallel File System (GPFS)

Overview

Every HPC node of both LiCCA and ALCC has access to the same network filesystem /hpc/gpfs2 which is a shared ressource.

This filesystem contains the following folders, which currently share the same performance characteristics:

  • User home directory /hpc/gpfs2/home/u/$USER
  • User scratch directory /hpc/gpfs2/scratch/u/$USER
  • Group home directory  /hpc/gpfs2/home/g/$HPC-Projekt/

  • Group scratch directory /hpc/gpfs2/scratch/g/$HPC-Projekt/ 

Backup

All content of /hpc/gpfs2/home is backed up once a day to the Tape Library of the Rechenzentrum. All important data (e.g. results of calculations, user maintained software, etc.) is recommended to be stored in User home or Group directories.

Pro Tip: All data that can easily be recreated (e.g. temporary files, python evironments, etc.) should be stored in the User scratch directory (not part of the Backup).

Default Permissions and Ownerships for User and Group directories

Once Project and Cluster access have been approved, default permissions as well as user and group ownerships are applied to the four directories listed above. Permissions and ownerships of existing files and folders in these directories remain untouched.

User directories

  • Owner: personal user account name
  • Group: generic user group with only the above owner as a member
  • Permissions: 0750
  • No additional ACL (Access Control Lists)

These directories can only be accessed by the owner and nobody else (except the root user). Default permissions of newly created files and folders are 0644  and 0755 , respectively, due to the default umask setting of 0022 . This does not mean that other cluster user may access your files, because no regular user can get past your personal home and scratch directories, which act as gatekeepers.

Group/Project directories

  • Owner: root
  • Group: root
  • Permissions: 0750
  • Additional ACL
ACL
#NFSv4 ACL
#owner:root
#group:root
special:owner@:rwxc:allow
 (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED     \
 (-)DELETE    (X)DELETE_CHILD (X)CHOWN        (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED    |
                                                                                                          |
special:group@:r-x-:allow                                                                                 |
 (X)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED     |\ Standard 0750 permissions
 (-)DELETE    (-)DELETE_CHILD (-)CHOWN        (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED    |/ for the root user
                                                                                                          |
special:everyone@:----:allow                                                                              |
 (-)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED     |
 (-)DELETE    (-)DELETE_CHILD (-)CHOWN        (-)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED    /

special:owner@:rwxc:allow:FileInherit:DirInherit:InheritOnly                                              \
 (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED     |
 (-)DELETE    (X)DELETE_CHILD (X)CHOWN        (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED    |
                                                                                                          |> ACL inherited by user created files and folders
group:rzhpc-<group>:rwxc:allow:FileInherit:DirInherit                                                     |  (does not apply to the group folder itself)
 (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED     |
 (X)DELETE    (X)DELETE_CHILD (X)CHOWN        (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED    /

These directories can (only) be accessed and modified by all group members. Files and directories created by one member can be aribrarily modified or removed by any other group member.

Note that user created files and folders in group directories won't have ACL an entry for special group  and other (everyone) permissions, therefore the last two mode bits (e.g. 700) or corresponding output of ls -l (e.g. -rwx------ ) is completely meaningless.

DO NOT attempt to "fix" file and folder permissions in group directories. Especially DO NOT run any kind of recursive chmod in group folders (e.g. chown -R ), even if you know what you are doing, because it is not necessary at all and will allocate useless extra metadata for every single file and folder.

Due to the nature of these ACL on group home and scratch directories, all files are marked as executable, and the output of ls may show all files with green color. Again, no need to fix this.

Granting Access to User and Group directories

User directories

DO NOT make your home or scratch folder world writable (e.g. chmod 777 ). This is explicitly forbidden and users doing so will receive a formal warning.

To grant readonly access for your home and/or scratch directory to a specific group:

Add an ACL entry for a rzhpc-* group
mmeditacl /hpc/gpfs2/home/u/$USER
- or -
mmeditacl /hpc/gpfs2/scratch/u/$USER

# Then append the following content and replace IDMGROUP with an existing rzhpc-* group:
group:IDMGROUP:r-x-:allow
 (X)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
 (-)DELETE    (-)DELETE_CHILD (-)CHOWN        (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED

The IdM group of choice should contain as few people as possible, because all members of this group will have read access to your personal home or scratch space this way. Recommendation: the respective rzhpc-* group of your project.

To grant readonly access for your home and/or scratch directory to a specific user:

Add an ACL entry for a single user
mmeditacl /hpc/gpfs2/home/u/$USER
- or -
mmeditacl /hpc/gpfs2/scratch/u/$USER

# Then append the following content and replace RZBK with the actual RZ user ID:
user:RZBK:r-x-:allow
 (X)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
 (-)DELETE    (-)DELETE_CHILD (-)CHOWN        (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED

Group/Project directories

You cannot modify the ACL of home and scratch group/project directories. To get access to another group's home or scratch folder you have to apply for Access to the Project Membership.