HPC/Sharing Files - Revision history

Stern at 15:25, April 28, 2025

2025-04-28T15:25:17Z

← Older revision		Revision as of 15:25, April 28, 2025
Line 8:		Line 8:
	See [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS#File transfer]].		See [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS#File transfer]].

	== ~~Group types~~ ==		== Shared group directories ==

	=== Proposal groups ===		=== Group types ===
			==== Proposal groups ====
	Typically, a group and associated sharing directory are created automatically for each '''CNM proposal''' that has more than one user.		Typically, a group and associated sharing directory are created automatically for each '''CNM proposal''' that has more than one user.
	Access those directories under one of the following names:		Access those directories under one of the following names:
Line 17:		Line 18:
	: Of course, replace ''12345'' by the relevant proposal that you are a member of.		: Of course, replace ''12345'' by the relevant proposal that you are a member of.

	=== Named project groups ===		==== Named project groups ====
	Leveraging a named ''project group'' is a convenient way for users to share files over periods longer than a single proposal will be active (12 to 18 months).		Leveraging a named ''project group'' is a convenient way for users to share files over periods longer than a single proposal will be active (12 to 18 months).

Line 28:		Line 29:
	# A ''user list'' of initial members; later additions or removals will be easy to make.		# A ''user list'' of initial members; later additions or removals will be easy to make.

	== ~~Using shared group directories~~ ==		=== Directory location ===
	~~=== Location~~ ===
	* A new Unix ''group'' will be created and populated by the relevant users as described above. Group members will receive an email with instructions that new group memberships will be "picked up" by the system upon the next login.		* A new Unix ''group'' will be created and populated by the relevant users as described above. Group members will receive an email with instructions that new group memberships will be "picked up" by the system upon the next login.
	* A group ''directory'' will be created at:		* A group ''directory'' will be created at:

Stern: /* Advanced permission handling */

2025-04-28T15:23:36Z

Advanced permission handling

← Older revision		Revision as of 15:23, April 28, 2025
Line 62:		Line 62:
	=== Advanced permission handling ===		=== Advanced permission handling ===
	Directories under <code>/home/SHARE</code> are typically created administratively		Directories under <code>/home/SHARE</code> are typically created administratively
	with the [https://en.wikipedia.org/wiki/Setuid#~~setuid_and_setgid_on_directories~~ setgid bit] active,		with the [https://en.wikipedia.org/wiki/Setuid#When_set_on_a_directory setgid bit] active,
	which will cause newly created files and directories to inherit the shared group of the parent, and thus become shared as well.		which will cause newly created files and directories to inherit the shared group of the parent, and thus become shared as well.
	But this inheritance does not apply to existing directories that have been moved here.		But this inheritance does not apply to existing directories that have been moved here.

Stern: /* Using shared group directories */

2025-04-28T15:22:49Z

Using shared group directories

← Older revision		Revision as of 15:22, April 28, 2025
Line 60:		Line 60:
	Lean more about [http://linuxfrombeginning.wordpress.com/2008/09/25/linux-command-9-chown-chgrp-chmod/ chmod, chown, and chgrp].		Lean more about [http://linuxfrombeginning.wordpress.com/2008/09/25/linux-command-9-chown-chgrp-chmod/ chmod, chown, and chgrp].

	~~'''~~Advanced~~:'''~~ Directories under <code>/home/SHARE</code> are typically created administratively		=== Advanced permission handling ===
			Directories under <code>/home/SHARE</code> are typically created administratively
	with the [https://en.wikipedia.org/wiki/Setuid#setuid_and_setgid_on_directories setgid bit] active,		with the [https://en.wikipedia.org/wiki/Setuid#setuid_and_setgid_on_directories setgid bit] active,
	which will cause newly created files and directories to inherit the shared group of the parent, and thus become shared as well.		which will cause newly created files and directories to inherit the shared group of the parent, and thus become shared as well.

Stern: /* Using shared group directories */

2025-04-28T15:22:21Z

Using shared group directories

← Older revision		Revision as of 15:22, April 28, 2025
Line 29:		Line 29:

	== Using shared group directories ==		== Using shared group directories ==
			=== Location ===
	* A new Unix ''group'' will be created and populated by the relevant users as described above. Group members will receive an email with instructions that new group memberships will be "picked up" by the system upon the next login.		* A new Unix ''group'' will be created and populated by the relevant users as described above. Group members will receive an email with instructions that new group memberships will be "picked up" by the system upon the next login.
	* A group ''directory'' will be created at:		* A group ''directory'' will be created at:
Line 59:		Line 60:
	Lean more about [http://linuxfrombeginning.wordpress.com/2008/09/25/linux-command-9-chown-chgrp-chmod/ chmod, chown, and chgrp].		Lean more about [http://linuxfrombeginning.wordpress.com/2008/09/25/linux-command-9-chown-chgrp-chmod/ chmod, chown, and chgrp].

	'''Advanced:''' Directories under /home/SHARE are typically created administratively		'''Advanced:''' Directories under <code>/home/SHARE</code> are typically created administratively
	with the [https://en.wikipedia.org/wiki/Setuid#setuid_and_setgid_on_directories setgid bit] active,		with the [https://en.wikipedia.org/wiki/Setuid#setuid_and_setgid_on_directories setgid bit] active,
	which will cause newly created files and directories to inherit the shared group of the parent, and thus become shared as well.		which will cause newly created files and directories to inherit the shared group of the parent, and thus become shared as well.

Stern: /* Managing permissions */

2025-04-28T15:21:25Z

Managing permissions

← Older revision		Revision as of 15:21, April 28, 2025
Line 48:		Line 48:
	* Unfortunately, files ''moved'' into a shared directory do not automatically become group-shared because files being moved retain their original typically user-private user and group ownerships.		* Unfortunately, files ''moved'' into a shared directory do not automatically become group-shared because files being moved retain their original typically user-private user and group ownerships.

	To share directories and files that are not accessible by group members, the ''owner'' ~~(or~~ an administrator) must run the following:		To share directories and files that are not accessible by group members, the ''file owner'' – or an administrator – must run the following:

	* To make files writeable by group members:		* To make files writeable by group members:

Stern: /* Managing permissions */

2025-04-28T15:21:07Z

Managing permissions

← Older revision		Revision as of 15:21, April 28, 2025
Line 45:		Line 45:
	Carbon's <code>/home/SHARE</code> directories are configured as follows:		Carbon's <code>/home/SHARE</code> directories are configured as follows:
	* The shared directory will be mutually readable and writable among group members, and will not be accessible by other users.		* The shared directory will be mutually readable and writable among group members, and will not be accessible by other users.
	* Files newly created in ~~the~~ shared directory will automatically become group-shared (by inheriting group membership). Unfortunately, ~~this does not apply for~~ files ''moved'' ~~there from somewhere else~~.		* Files ''newly created'' in a shared directory will automatically become group-shared (by inheriting group membership).
			* Unfortunately, files ''moved'' into a shared directory do not automatically become group-shared because files being moved retain their original typically user-private user and group ownerships.

	To share directories and files that are not accessible by group members, the ''owner'' (or an administrator) must run the following:		To share directories and files that are not accessible by group members, the ''owner'' (or an administrator) must run the following:

Stern: /* Using shared group directories */

2025-04-28T15:17:50Z

Using shared group directories

← Older revision		Revision as of 15:17, April 28, 2025
Line 41:		Line 41:
	** Use relative path names, such as <code>''dir/under/here''</code> or <code>../''otherdir''/''other_subdir/''</code>		** Use relative path names, such as <code>''dir/under/here''</code> or <code>../''otherdir''/''other_subdir/''</code>
	** Use absolute paths only when other directories are needed as well.		** Use absolute paths only when other directories are needed as well.

			=== Managing permissions ===
			Carbon's <code>/home/SHARE</code> directories are configured as follows:
	* The shared directory will be mutually readable and writable among group members, and will not be accessible by other users.		* The shared directory will be mutually readable and writable among group members, and will not be accessible by other users.
	* ~~The Unix~~ group ~~property will usually be inherited~~ by ~~all~~ files ~~and directories created~~ there.		* Files newly created in the shared directory will automatically become group-shared (by inheriting group membership). Unfortunately, this does not apply for files ''moved'' there from somewhere else.

	~~=== Managing permissions ===~~		To share directories and files that are not accessible by group members, the ''owner'' (or an administrator) must run the following:
	~~Carbon's <code>/home/SHARE</code> directories are configured such that users of a group can read and write files and directories.~~
	~~<!-- from somewhere else into the shared directory they may not be -->~~
	To share directories and files that are ~~for some reason~~ not accessible by group members, the ''owner'' (or an administrator) must run the following:

	* To make files writeable by group members:		* To make files writeable by group members:

Stern: /* Using shared group directories */

2025-04-28T15:13:48Z

Using shared group directories

← Older revision		Revision as of 15:13, April 28, 2025
Line 37:		Line 37:
	~cnm''12345''		~cnm''12345''
	: This "<code>~</code>" shortcut form is typically not interpreted in application programs (written in Fortran, C, Python, etc.). To best use shared directories with such applications, best practices are:		: This "<code>~</code>" shortcut form is typically not interpreted in application programs (written in Fortran, C, Python, etc.). To best use shared directories with such applications, best practices are:
	** ~~issue~~ a <code>cd</code> shell command before running the program, or		** Issue a <code>cd</code> shell command before running the program, or
	** in the calling shell or job script, set and export environment variables if supported by the application. They often are of the form <code>''FOO''_HOME</code>, <code>''FOO''_ROOT</code>, or <code>''FOO''_LIBDIR</code>. Carbon's "module" scripts for applications automatically do this.		** In the calling shell or job script, set and export environment variables if supported by the application. They often are of the form <code>''FOO''_HOME</code>, <code>''FOO''_ROOT</code>, or <code>''FOO''_LIBDIR</code>. Carbon's "module" scripts for applications automatically do this.
	** ~~use~~ relative path names, such as <code>''dir/under/here''</code> or <code>../''otherdir''/''other_subdir/''</code>		** Use relative path names, such as <code>''dir/under/here''</code> or <code>../''otherdir''/''other_subdir/''</code>
	** ~~use~~ absolute paths only when other directories are needed as well.		** Use absolute paths only when other directories are needed as well.
	* The directory will be mutually readable and writable among group members, and will not be accessible by other users.		* The shared directory will be mutually readable and writable among group members, and will not be accessible by other users.
	* The group will usually be inherited by all files and directories created there.		* The Unix group property will usually be inherited by all files and directories created there.

	=== Managing permissions ===		=== Managing permissions ===

Stern: /* Using shared group directories */

2025-04-28T15:12:46Z

Using shared group directories

← Older revision		Revision as of 15:12, April 28, 2025
Line 38:		Line 38:
	: This "<code>~</code>" shortcut form is typically not interpreted in application programs (written in Fortran, C, Python, etc.). To best use shared directories with such applications, best practices are:		: This "<code>~</code>" shortcut form is typically not interpreted in application programs (written in Fortran, C, Python, etc.). To best use shared directories with such applications, best practices are:
	** issue a <code>cd</code> shell command before running the program, or		** issue a <code>cd</code> shell command before running the program, or
	** in the calling shell or job script, set and export environment variables if supported by the application. They often are of the form <code>''FOO''_HOME</code>, <code>''FOO''_ROOT</code>, or <code>''FOO''_LIBDIR</code>. Carbon's "module"~~-based~~ applications automatically do this ~~for applications~~.		** in the calling shell or job script, set and export environment variables if supported by the application. They often are of the form <code>''FOO''_HOME</code>, <code>''FOO''_ROOT</code>, or <code>''FOO''_LIBDIR</code>. Carbon's "module" scripts for applications automatically do this.
	** use relative path names, such as <code>''dir/under/here''</code> or <code>../''otherdir''/''other_subdir/''</code>		** use relative path names, such as <code>''dir/under/here''</code> or <code>../''otherdir''/''other_subdir/''</code>
	** use absolute paths only when other directories are needed as well.		** use absolute paths only when other directories are needed as well.

Stern: /* Using shared group directories */

2025-04-28T15:12:21Z

Using shared group directories

← Older revision		Revision as of 15:12, April 28, 2025
Line 36:		Line 36:
	~''groupname''		~''groupname''
	~cnm''12345''		~cnm''12345''
	: This "<code>~</code>" shortcut form is typically not interpreted in application programs (written in Fortran, C, Python, etc.). To best use shared directories with such applications, best ~~practice is~~:		: This "<code>~</code>" shortcut form is typically not interpreted in application programs (written in Fortran, C, Python, etc.). To best use shared directories with such applications, best practices are:
	** issue a <code>cd</code> shell command before running the program, and ~~then~~		** issue a <code>cd</code> shell command before running the program, or
	** use relative path names, such as <code>''dir/under/here''</code> or <code>../''otherdir''/''other_subdir/''</code>.		** in the calling shell or job script, set and export environment variables if supported by the application. They often are of the form <code>''FOO''_HOME</code>, <code>''FOO''_ROOT</code>, or <code>''FOO''_LIBDIR</code>. Carbon's "module"-based applications automatically do this for applications.
			** use relative path names, such as <code>''dir/under/here''</code> or <code>../''otherdir''/''other_subdir/''</code>
	** use absolute paths only when other directories are needed as well.		** use absolute paths only when other directories are needed as well.
	* The directory will be mutually readable and writable among group members, and will not be accessible by other users.		* The directory will be mutually readable and writable among group members, and will not be accessible by other users.