By Robert Stober | September 26, 2012 | head node, Directory export, HPC nodes
This article describes how to NFS export a directory from the head node and mount it on a set of nodes using the Bright Cluster Manager CMSH. It only takes 8 easy steps to complete and verify this task.
We'll use /tmp as an example. Here's what we need to do:
1. NFS export /scratch on the head node
2. Mount master:/tmp on the nodes as /mnt/tmp
Let's get started.
First, let's verify that the mount point /mnt/tmp does not exist on the nodes in the default node category.
[atom-head1->device]% pexec -c default -j "ls -l /mnt"
Nodes down: atomgpu01
[atomgpu01]
Node down
[atom01..atom03]
total 0
Next, Export the /tmp directory from the head node. Note that the name /tmp@internal is used by convention.
atom-head1]% device use master
[atom-head1->device[atom-head1]]% fsexports
[atom-head1->device[atom-head1]->fsexports]% add /tmp@internalnet
[atom-head1->device*[atom-head1*]->fsexports*[/tmp@internalnet*]]% set path /tmp
[atom-head1->device*[atom-head1*]->fsexports*[/tmp@internalnet*]]% set write true
[atom-head1->device*[atom-head1*]->fsexports*[/tmp@internalnet*]]% set async true
[atom-head1->device*[atom-head1*]->fsexports*[/tmp@internalnet*]]% set fsid 5
[atom-head1->device*[atom-head1*]->fsexports*[/tmp@internalnet*]]% set hosts internalnet
[atom-head1->device*[atom-head1*]->fsexports*[/tmp@internalnet*]]% commit
Verify the directory has been exported:
[atom-head1->device[atom-head1]->fsexports[/tmp@internalnet]]% show
Parameter Value
------------------------------ ------------------------------------------------
All squash no
Anonymous GID 65534
Anonymous UID 65534
Async yes
Extra options
Hosts internalnet (10.141.0.0/16)
Name /tmp@internalnet
Path /tmp
RDMA no
Revision
Root squash no
Write yes
fsid 5
Also verify using the standard 'exportfs' command:
[root@atom-head1 default-image]# exportfs
/cm/shared 10.141.0.0/16
/home 10.141.0.0/16
/var/spool/burn 10.141.0.0/16
/cm/node-installer/certificates 10.141.0.0/16
/cm/node-installer 10.141.0.0/16
/tmp 10.141.0.0/16
Now let's add the mount to the default node category. Note that the device name $localnfsserver resolves to the local NFS server, which could be the head node (in the case of local nodes) or the cloud director (in the case of cloud nodes).
[atom-head1->category[default]->fsmounts]% add /mnt/tmp
[atom-head1->category*[default*]->fsmounts*[/mnt/tmp*]]% set device $localnfsserver:/tmp
[atom-head1->category*[default*]->fsmounts*[/mnt/tmp*]]% set mountpoint /mnt/tmp
[atom-head1->category*[default*]->fsmounts*[/mnt/tmp*]]% set filesystem nfs
[atom-head1->category*[default*]->fsmounts*[/mnt/tmp*]]% commit
[atom-head1->category[default]->fsmounts[/mnt/tmp]]% list
Device Mountpoint (key) Filesystem
--------------------------- -------------------------------- ----------------
none /dev/pts devpts
none /proc proc
none /sys sysfs
none /dev/shm tmpfs
$localnfsserver:/cm/shared /cm/shared nfs
$localnfsserver:/home /home nfs
$localnfsserver:/tmp /mnt/tmp nfs
Let's verify that we're done:
[atom-head1->device]% pexec -c default -j mount
Nodes down: atomgpu01
[atomgpu01]
Node down
[atom01..atom03]
rootfs on / type rootfs (rw)
proc on /proc type proc (rw,relatime)
sys on /sys type sysfs (rw,relatime)
tmpfs on / type tmpfs (rw,relatime,size=0k,mpol=interleave:0)
/sysroot/proc on /proc type proc (rw,relatime)
/sysroot/sys on /sys type sysfs (rw,relatime)
/proc/bus/usb on /proc/bus/usb type usbfs (rw,relatime)
none on /dev type devtmpfs (rw,relatime,size=2017428k,nr_inodes=504357,mode=755)
none on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
none on /dev/shm type tmpfs (rw,relatime)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
master:/cm/shared on /cm/shared type nfs (rw,relatime)
master:/home on /home type nfs (rw,relatime)
master:/tmp on /mnt/tmp type nfs (rw,relatime)
Now let's list the /mnt/tmp directory on the cluster nodes:
[atom-head1->device]% pexec -c default -j "ls -l /mnt"
Nodes down: atomgpu01
[atomgpu01]
Node down
[atom01..atom03]
total 4
drwxr-xr-x 10 2402 gopher 4096 Aug 8 12:15 tmp
And now the /mnt/tmp directory:
[atom-head1->device]% pexec -c default -j "ls -l /mnt/tmp"
Nodes down: atomgpu01
[atomgpu01]
Node down
[atom01..atom03]
total 15512
drwxr-xr-x 2 root root 4096 Jul 31 22:57 arcetri
-rw-r--r-- 1 root root 13604922 Jul 7 2011 LinuxKit.tgz
drwx------ 2 root root 16384 Jul 7 10:41 lost+found
drwx------ 2 42 42 4096 Jul 27 06:21 orbit-gdm
drwx------ 2 42 42 4096 Jul 27 06:21 pulse-lIAzUuyS51kt
drwx------ 2 root root 4096 Jul 27 06:16 pulse-xfs6ftI4dQkC
-rw-r--r-- 1 root root 629195 Aug 6 14:25 saved.xml
-rw------- 1 48 48 317799 Jul 31 09:19 wsdl-apache-86d9865ed335cf24a654453d08a343e0
-rw------- 1 48 48 308905 Jul 31 09:19 wsdl-apache-a2a536a5a8e2531de0452475ea8eafbd
-rw------- 1 48 48 305583 Jul 31 09:19 wsdl-apache-a3b2f8a80c872439779f10de02a4a56d
-rw------- 1 48 48 311556 Jul 31 09:19 wsdl-apache-d9e05d8b6c93ebbacad5c34a497807c6
-rw------- 1 48 48 314514 Jul 31 09:19 wsdl-apache-f0fe8bcff41e819b66bb9f66358195c0
We're done.