HPC/Sharing Files

Introduction

Sharing files on Carbon

When you collaborate with other users on a common project on Carbon you will naturally wish to share files among you. There are several ways to do this, largely distinguished by how often and for how long sharing is needed.

Transferring files to or from Carbon

See HPC/Network Access/SSH Tunnel Setup on Linux and MacOS#File transfer.

Sharing within a project group

Leveraging a group is a convenient way for users to share files repeatedly over an extended period of time.

To use this method, request a project group and directory by mail. Send the following information:

1. a reason for the request,
2. a group name which should be short (up to some 8 chars), descriptive, and distinctive,
3. the initial user list (later changes are easy).

• A new Unix group will be created and users will be made members of that group. To pick up such a change, log out and back in.
• A group directory will be created and can be accessed by one of the following names:

/home/share/groupname
~groupname

Note that Fortran programs typically cannot interpret the "~" shortcut form of the share directory. Either give the full path or cd into the directory before running the program, then use relative path names.

• The directory will be mutually readable and writable among group members, and will not be accessible by other users.
• The group will usually be inherited by all files and directories created there.

Sharing within proposals

Typically, a project group and associated sharing directory are created automatically for each CNM proposal that has more than one user. Access those directories under one of the following names:

/home/share/cnm12345
~cnm12345

Managing permissions

Carbon's /home/share directories are configured such that users of a group can read and write files and directories. To share directories and files that are for some reason not accessible by group members, the owner (or an administrator) must run the following:

• To make files writeable by group members:

chgrp groupname shared_file ...
chmod g+rw shared_file ...

• To make a directory and all files thereunder (-R) group-writeable, which includes the ability to delete files:

chgrp -R groupname shared_dir_name
chmod -R g+rwX shared_dir_name

Lean more about chmod, chown, and chgrp.

Advanced: Directories under /home/share are typically created administratively with the setgid bit active, which will cause newly created files and directories to inherit the shared group of the parent, and thus become shared as well. But this inheritance does not apply to existing directories that have been moved here.

To share moved directories, run the commands above, then switch on group inheritance as follows:

find shared_dir_name -type d -print0 | xargs -0 chmod -v g+s

The first part of this pipeline can be run alone to simply show a list of all directories (-type d), like so:

find shared_dir_name -type d
find shared_dir_name -type d  -ls

The full command as shown earlier produces that list in a robust manner (using NULL-terminated strings of names), then uses xargs(1) to run chmod(1) to activate the setgid bit on these directories (but not files).

Alternatively, instead of moving files into a proposal-shared directory, copy them first, then remove the source:

cp -R --preserve=timestamps  source_dir  ~cnm123456/
rm -r source_dir

Sharing in your own directories

You can give access to some of your directories in your respective $HOME or $SANDBOX by opening permissions.

• This option is typically useful for one-way sharing.
• For bidirectional sharing, you will need to grant world-write permissions, which is not recommended for files in your own directory space.
• You will likely need to repeatedly open permissions (chmod) for additional files that are placed there.

You as the owner can arrange this on your own as follows:

chmod a+x $HOME/
mkdir $HOME/share
# populate ..
chmod -R a+rX $HOME/share

This will yield read ("r") and execute ("X") permissions for all users ("a"), meaning users can browse directories, and read and execute already executable files there.

You may have to repeat the last step after you add files in the shared directory.

Direct your fellow users as follows:

cd /home/owner/share
ls
cp -p ....

The first cd is crucial to have your collaborator step into a sharing directory you designated instead of stepping in from your home directory. Other users are not usually able to simply rummage around your home directory with "ls" – it is up to you to show them a specific directory they can browse.

If you have particularly large files, replace $HOME and /home by $SANDBOX and /sandbox, respectively.

Using a /tmp directory

You can drop files into the general-use /tmp directory of a login node and set permissions as needed. This way of sharing is:

• suitable for one-off exchanges,
• typically fairly loose with permissions,
• resides on only one host,
• world-write is not recommended,
• likely to need chmod whenever more files are placed.

To share files under /tmp, do the following:

mkdir /tmp/foo
# populate ..
chmod -R a+rX /tmp/foo