▷ How to run Linpack across a Bright Linux cluster

It's easy to run Linpack across a Linux cluster with Bright Cluster Manager. This article shows how, using the Bright CMSH.

1. Set up your build environment

# cd /tmp
# module purge
# module add shared openmpi/gcc gotoblas

2. Build the RPM, this creates the patched sources and modified Makefile

# rpmbuild -bb /usr/src/redhat/SPECS/hpl.spec

B. Now log in (or su) as an ordinary user and perform the rest of the commands as this user.

1. Prepare the source

# su - rstober
$ cp -a /usr/src/redhat/BUILD/hpl-2.0 .
$ cd hpl-2.0
$ mv Make.Linux_OPTERON_GOTOBLAS64 Make.`arch`

2. Choose your environment

Note: In the near future you will want to install and use the Intel compiler as you will get a signifcantly better Linpack result.

$ module purge
$ module add openmpi/gcc

Change the ARCH and LINKER parameters (if necessary) to the values shown below

$ vi Make.x86_64

ARCH         = x86_64
LINKER       = mpicc

3. Use GotoBLAS

Note: gotoblas has optimization for both intel and amd cpu's make sure you choose the correct module as this can improve your results with 50%

$ module add gotoblas/penryn

Change the LAdir and LAlib parameters (if necessary) to the values shown below.

$ vi Make.x86_64

LAdir        = $(BLASDIR)
LAlib        = -L$(LAdir) -l$(BLASLIB)

4. Build the xhpl binary

$ TOPdir=`pwd` make arch=x86_64

Now copy the input file and binary to your running directory

$ cd ~
$ cp hpl-2.0/bin/x86_64/HPL.dat .
$ cp hpl-2.0/bin/x86_64/xhpl .

5. Run HPL across your cluster

Create a machines file

$ cat machines
atom01
atom02
atom03

Add the required modules to your environment. This ensures that the execution environment is correct in the remote shells that mpirun spawns.

$ module initadd openmpi/gcc
$ module initadd gotoblas

Run xhpl across the cluster

$ mpirun -np 12 -machinefile machines ./xhpl | tee HPL.out

C. Next Steps

1. Install the Intel compiler
2. Build xhpl using the Intel compiler
3. Tune the HPL.dat file to maximize performance