HPC/Sharing Files

Introduction

Sharing files on Carbon

When you collaborate with other users on a common project on Carbon you will naturally wish to share files among you. There are several ways to do this, largely distinguished by how often and for how long sharing is needed.

Transferring files to or from Carbon

See HPC/Network Access/SSH Tunnel Setup on Linux and MacOS#File transfer.

Sharing within a project group

Leveraging a group is a convenient way for users to share files repeatedly over an extended period of time.

To use this method, request a project group and directory by mail. Send the following information:

1. a reason for the request,
2. a group name which should be short (up to some 8 chars), descriptive, and distinctive,
3. the initial user list (later changes are easy).

• A new Unix group will be created and users will be made members of that group. To pick up such a change, log out and back in.
• A group directory will be created and can be accessed by one of the following names:

/home/share/groupname
~groupname

Note that Fortran programs typically cannot interpret the "~" shortcut form of the share directory. Either give the full path or cd into the directory before running the program, then use relative path names.

• The directory will be mutually readable and writable among group members, and will not be accessible by other users.
• The group will usually be inherited by all files and directories created there.

Sharing within proposals

Typically, a project group and associated sharing directory are created automatically for each CNM proposal that has more than one user. Access those directories under one of the following names:

/home/share/cnm12345
~cnm12345

Giving write permission

Carbon's /home/share directories are configured such that users of a group can read and write files and directories. However, when you move existing directories and files from somewhere else into the shared directory they may not be writeable for the group.

• To make files writeable by group members:

chgrp groupname file ...
chmod g+rw file ...

• To make a directory and all files thereunder (-R) group-writeable, which includes the ability to delete files:

chgrp -R groupname dirname
chmod -R g+rwX dirname

Lean more about chmod, chown, and chgrp.

Advanced: Directories under /home/share are typically created with the setgid bit active, which causes files and directories to inherit the shared group ownership. This inheritance applies only to newly created files or dirs. To switch on group inheritance for existing directories that have been moved into a shard directory, run the following additional command after the ones above:

find dirname -type d -print0 | xargs -0 chmod -v g+s

The first part of this pipeline can be run alone to simply show a list of all directories (-type d), like so:

find dirname -type d
find dirname -type d  -ls

The full command produces the list in a robust manner (NULL-terminated), then uses xargs(1) to run chmod(1), thereby affecting all child directories (but not files).

Sharing in your own directories

You can give access to some of your directories in your respective $HOME or $SANDBOX by opening permissions.

• This option is typically useful for one-way sharing.
• For bidirectional sharing, you will need to grant world-write permissions, which is not recommended for files in your own directory space.
• You will likely need to repeatedly open permissions (chmod) for additional files that are placed there.

You as the owner can arrange this on your own as follows:

chmod a+x $HOME/
mkdir $HOME/share
# populate ..
chmod -R a+rX $HOME/share

This will yield read ("r") and execute ("X") permissions for all users ("a"), meaning users can browse directories, and read and execute already executable files there.

You may have to repeat the last step after you add files in the shared directory.

Direct your fellow users as follows:

cd /home/owner/share
ls
cp -p ....

The first cd is crucial to have your collaborator step into a sharing directory you designated instead of stepping in from your home directory. Other users are not usually able to simply rummage around your home directory with "ls" – it is up to you to show them a specific directory they can browse.

If you have particularly large files, replace $HOME and /home by $SANDBOX and /sandbox, respectively.

Using a /tmp directory

You can drop files into the general-use /tmp directory of a login node and set permissions as needed. This way of sharing is:

• suitable for one-off exchanges,
• typically fairly loose with permissions,
• resides on only one host,
• world-write is not recommended,
• likely to need chmod whenever more files are placed.

To share files under /tmp, do the following:

mkdir /tmp/foo
# populate ..
chmod -R a+rX /tmp/foo