HPC/Sharing Files: Difference between revisions

From CNM Wiki
< HPC
Jump to navigation Jump to search
 
(30 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Introduction ==
== Introduction ==


When you collaborate with other users on a common project on Carbon, you may wish to share files among you. There are several ways to achieve this, largely distinguished by how often and for how long sharing is needed.
=== Sharing files ''on'' Carbon ===
When you collaborate with other users on a common project on Carbon you will naturally wish to share files among you.
There are several ways to do this, largely distinguished by how often and for how long sharing is needed.


== Sharing Types ==
=== Transferring files ''to or from'' Carbon ===
See [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS#File transfer]].


=== Using a Unix group ===
== Sharing within a project group ==
Leveraging a ''group'' is a convenient way for users to share files repeatedly over an extended period of time.


* This option is suitable for ongoing exchanges.
To use this method, [mailto:[email protected]?Subject=Carbon%20HPC%20Request%3a%20ad-hoc%20group request a project group and directory] by mail. Send the following information:
# a reason for the request,
# a ''group name'' which should be short (up to some 8 chars), descriptive, and distinctive,
# the initial ''user list'' (later changes are easy).
 
* A new Unix group will be created and users will be made members of that group. To pick up such a change, log out and back in.
* A group directory will be created and can be accessed by one of the following names:
* A group directory will be created and can be accessed by one of the following names:
  /home/share/''groupname''
  /home/share/''groupname''
  ~''groupname''
  ~''groupname''
: Note that Fortran programs typically cannot interpret the "<code>~</code>" shortcut form of the share directory. Either give the full path or cd into the directory before running the program, then use relative path names.
* The directory will be mutually readable and writable among group members, and will not be accessible by other users.
* The directory will be mutually readable and writable among group members, and will not be accessible by other users.
* Permissions will be inherited to all files placed there.
* The group will usually be inherited by all files and directories created there.
Typically, such a group name and directory is created automatically for each CNM proposal that has more than one user. You can access the shared directories for proposals by one of the following:
 
=== Sharing within proposals ===
Typically, a project group and associated sharing directory are created automatically for each CNM proposal that has more than one user.
Access those directories under one of the following names:
  /home/share/cnm''12345''
  /home/share/cnm''12345''
  ~cnm''12345''
  ~cnm''12345''
The '''''system admin''''' can create a project group and directory for you.  To [mailto:[email protected]?Subject=ad-hoc%20group request the creation of a group], please send the following information:
# a ''group name'' which should be short (up to some 8 chars), descriptive, and distinctive.
# the ''user list'' (later changes are easy).


=== Sharing in your own directories ===
=== Managing permissions ===
Carbon's /home/share directories are configured such that users of a group can read and write files and directories.
<!--  from somewhere else into the shared directory they may not be -->
To share directories and files that are for some reason not accessible by group members, the ''owner'' (or an administrator) must run the following:
 
* To make files writeable by group members:
chgrp ''groupname'' ''shared_file'' ...
chmod g+rw ''shared_file'' ...
* To make a directory and all files thereunder (-R) group-writeable, which includes the ability to ''delete'' files:
chgrp -R ''groupname'' ''shared_dir_name''
chmod -R g+rwX ''shared_dir_name''
 
Lean more about [http://linuxfrombeginning.wordpress.com/2008/09/25/linux-command-9-chown-chgrp-chmod/ chmod, chown, and chgrp].
 
'''Advanced:''' Directories under /home/share are typically created administratively
with the [https://en.wikipedia.org/wiki/Setuid#setuid_and_setgid_on_directories setgid bit] active,
which will cause newly created files and directories to inherit the shared group of the parent, and thus become shared as well.
But this inheritance does not apply to existing directories that have been moved here.
 
To share ''moved'' directories, run the commands above, then switch on group inheritance as follows:
find ''shared_dir_name'' -type d -print0 | xargs -0 chmod -v g+s
The first part of this pipeline can be run alone to simply show a list of all directories (<code>-type d</code>), like so:
find ''shared_dir_name'' -type d
find ''shared_dir_name'' -type d  -ls
The full command as shown earlier produces that list in a robust manner (using NULL-terminated strings of names), then uses [http://man7.org/linux/man-pages/man1/xargs.1.html xargs(1)] to run [http://man7.org/linux/man-pages/man1/chmod.1.html chmod(1)] to activate the setgid bit on these directories (but not files).
 
Alternatively, instead of ''moving'' files into a proposal-shared directory, ''copy'' them first, then ''remove'' the source:
 
cp -R --preserve=timestamps  ''source_dir''  ~cnm123456/
rm -r ''source_dir''
 
== Sharing in your own directories ==


You can give access to some of your directories in [[../Directories | your respective $HOME or $SANDBOX]] by opening permissions.
You can give access to some of your directories in [[../Directories | your respective $HOME or $SANDBOX]] by opening permissions.


* This option is typically useful for one-way sharing.
* This option is typically useful for one-way sharing.
* For bidirectional sharing, you will need to grant world-write permissions, which is not recommended.
* For bidirectional sharing, you will need to grant world-write permissions, which is not recommended for files in your own directory space.
* You will likely need to repeatedly open permissions (<code>chmod</code>) for additional files that are placed there.
* You will likely need to repeatedly open permissions (<code>chmod</code>) for additional files that are placed there.


'''''You''''' can arrange this on your own as follows:
You as the '''owner''' can arrange this on your own as follows:
 
; Owner:
  chmod a+x $HOME/
  chmod a+x $HOME/
  mkdir $HOME/share
  mkdir $HOME/share
  # populate ..
  # populate ..
  chmod -R a+rX $HOME/share
  chmod -R a+rX $HOME/share
: This will yield ''read'' ("r") and ''execute'' ("X") permissions for all users ("a"), meaning users can browse directories, and read and execute already executable files there.
: You may have to repeat the last step after you add files in the shared directory.
: You may have to repeat the last step after you add files in the shared directory.
; Reader:
 
  cd /home/owner/share
Direct your '''fellow users''' as follows:
  cd /home/''owner''/share
  ls
  ls
  cp -p ....
  cp -p ....
The first cd is crucial to have your collaborator step into a sharing directory you designated instead of stepping in from your home directory.
Other users are not usually able to simply rummage around your home directory with "ls" – it is up to you to show them a specific directory they can browse.


If you have particularly large files, replace <code>$HOME</code> and <code>/home</code> by <code>$SANDBOX</code> and <code>/sandbox</code>, respectively.
If you have particularly large files, replace <code>$HOME</code> and <code>/home</code> by <code>$SANDBOX</code> and <code>/sandbox</code>, respectively.


=== Using a /tmp directory ===
== Using a /tmp directory ==
 
You can drop files into the general-use /tmp directory of a login node and set permissions as needed. This way of sharing is:
* suitable for one-off exchanges
* suitable for one-off exchanges,
* typically fairly loose permissions
* typically fairly loose with permissions,
* resides on only one host
* resides on only one host,
* world-write not recommended
* world-write is not recommended,
* likely to need chmod whenever more files are placed
* likely to need chmod whenever more files are placed.


'''''You''''' can arrange this on your own as well:
To share files under /tmp, do the following:
  mkdir /tmp/foo
  mkdir /tmp/foo
  # populate ..
  # populate ..
  chmod -R a+rX /tmp/foo
  chmod -R a+rX /tmp/foo

Latest revision as of 15:57, November 9, 2023

Introduction

Sharing files on Carbon

When you collaborate with other users on a common project on Carbon you will naturally wish to share files among you. There are several ways to do this, largely distinguished by how often and for how long sharing is needed.

Transferring files to or from Carbon

See HPC/Network Access/SSH Tunnel Setup on Linux and MacOS#File transfer.

Sharing within a project group

Leveraging a group is a convenient way for users to share files repeatedly over an extended period of time.

To use this method, request a project group and directory by mail. Send the following information:

  1. a reason for the request,
  2. a group name which should be short (up to some 8 chars), descriptive, and distinctive,
  3. the initial user list (later changes are easy).
  • A new Unix group will be created and users will be made members of that group. To pick up such a change, log out and back in.
  • A group directory will be created and can be accessed by one of the following names:
/home/share/groupname
~groupname
Note that Fortran programs typically cannot interpret the "~" shortcut form of the share directory. Either give the full path or cd into the directory before running the program, then use relative path names.
  • The directory will be mutually readable and writable among group members, and will not be accessible by other users.
  • The group will usually be inherited by all files and directories created there.

Sharing within proposals

Typically, a project group and associated sharing directory are created automatically for each CNM proposal that has more than one user. Access those directories under one of the following names:

/home/share/cnm12345
~cnm12345

Managing permissions

Carbon's /home/share directories are configured such that users of a group can read and write files and directories. To share directories and files that are for some reason not accessible by group members, the owner (or an administrator) must run the following:

  • To make files writeable by group members:
chgrp groupname shared_file ...
chmod g+rw shared_file ...
  • To make a directory and all files thereunder (-R) group-writeable, which includes the ability to delete files:
chgrp -R groupname shared_dir_name
chmod -R g+rwX shared_dir_name

Lean more about chmod, chown, and chgrp.

Advanced: Directories under /home/share are typically created administratively with the setgid bit active, which will cause newly created files and directories to inherit the shared group of the parent, and thus become shared as well. But this inheritance does not apply to existing directories that have been moved here.

To share moved directories, run the commands above, then switch on group inheritance as follows:

find shared_dir_name -type d -print0 | xargs -0 chmod -v g+s

The first part of this pipeline can be run alone to simply show a list of all directories (-type d), like so:

find shared_dir_name -type d
find shared_dir_name -type d  -ls

The full command as shown earlier produces that list in a robust manner (using NULL-terminated strings of names), then uses xargs(1) to run chmod(1) to activate the setgid bit on these directories (but not files).

Alternatively, instead of moving files into a proposal-shared directory, copy them first, then remove the source:

cp -R --preserve=timestamps  source_dir  ~cnm123456/
rm -r source_dir

Sharing in your own directories

You can give access to some of your directories in your respective $HOME or $SANDBOX by opening permissions.

  • This option is typically useful for one-way sharing.
  • For bidirectional sharing, you will need to grant world-write permissions, which is not recommended for files in your own directory space.
  • You will likely need to repeatedly open permissions (chmod) for additional files that are placed there.

You as the owner can arrange this on your own as follows:

chmod a+x $HOME/
mkdir $HOME/share
# populate ..
chmod -R a+rX $HOME/share
This will yield read ("r") and execute ("X") permissions for all users ("a"), meaning users can browse directories, and read and execute already executable files there.
You may have to repeat the last step after you add files in the shared directory.

Direct your fellow users as follows:

cd /home/owner/share
ls
cp -p ....

The first cd is crucial to have your collaborator step into a sharing directory you designated instead of stepping in from your home directory. Other users are not usually able to simply rummage around your home directory with "ls" – it is up to you to show them a specific directory they can browse.

If you have particularly large files, replace $HOME and /home by $SANDBOX and /sandbox, respectively.

Using a /tmp directory

You can drop files into the general-use /tmp directory of a login node and set permissions as needed. This way of sharing is:

  • suitable for one-off exchanges,
  • typically fairly loose with permissions,
  • resides on only one host,
  • world-write is not recommended,
  • likely to need chmod whenever more files are placed.

To share files under /tmp, do the following:

mkdir /tmp/foo
# populate ..
chmod -R a+rX /tmp/foo