Installing MPB on High Performance Computing Clusters

MPB Main Page: http://ab-initio.mit.edu/wiki/index.php/MIT_Photonic_Bands

In /usr/local/lib & /usr/local/include, check if you have the libraries and header files, respectively, for

• BLAS (libblas.a)
• LAPACK (liblapack.a)

Installing libctl

Reference: http://ab-initio.mit.edu/wiki/index.php/Libctl

Install FFTW

Reference: http://www.fftw.org/fftw3_doc/Installation-and-Customization.html

Important: you will also need to install the MPI FFTW libraries (this just means including --enable-mpi in the FFTW configure flags)

On SGI (Silicon Graphics) machine

export LDFLAGS="-L$HOME/scratch/fftw-3.3.4/lib -L/apps/HDF5/1.8.14/share/lib -L//apps/MPB/LIBCTL/3.2.2/lib"

export CPPFLAGS="-I$HOME/scratch/fftw-3.3.4/include -I/apps/HDF5/1.8.14/include -I/apps/MPB/LIBCTL/3.2.2/include"

export WM_CFLAGS="$WM_CFLAGS -DMPI_NO_CPPBIND -DSGIMPI"

gcc -lmpi

./configure --prefix=$HOME/scratch/mpb/ --with-mpi - --with-libctl=$HOME/scratch/libctl/libctl-3.2.2/share/libctl

On Fujitsu machine

export LDFLAGS="-L/apps/FFTW/3.3.4-gcc-4.8.2/lib -L"
export CPPFLAGS="-I/apps/FFTW/3.3.4-gcc-4.8.2/include -I"

Creating RAID 6 in Linux

Goal: 4x RAID 6 arrays
Each array will consist of 10 disks + 2 spares.

For the first array:

Use fdisk to partition the HDs
Active: /dev/sd[b-k]
Spare: /dev/sd[l-m]

Examine the disks
mdadm -E /dev/sd[b-k]1

Create RAID 6 array
mdadm --create /dev/md0 --level=6 --raid-devices=10 /dev/sd[b-k]1

Create 2 spare disks
mdadm --add /dev/md0 /dev/sdl1 /dev/sdm1

Sanity check
cat /proc/mdstat
mdadm -E /dev/sd[b-m]1
mdadm --detail /dev/md0

Do the same for the remaining 3 arrays. Remember to save the mdadm config by

mdadm --detail --scan --verbose >> /etc/mdadm.conf

LVM

vgcreate vg1 /dev/md0 /dev/md1 /dev/md2

lvcreate -l 100%FREE -n lv1 vg1

mkfs.xfs /dev/vg1/lv1

mount -t xfs /dev/vg1/lv1 /mnt/test


In case of one array fails.

lvremove /dev/vg1/lv1

vgreduce vg1 --removemissing

#Remove the offending array, e.g /dev/md0
mdadm --stop /dev/md0
mdadm --zero-superblock /dev/sd[b-m]1

#Recreate the RAID array.

#Then add to vg1
vgextend vg1 /dev/md0

#Recreate the volume group and mount.
-------------------
References

http://www.tecmint.com/create-raid-6-in-linux/

http://www.thegeekstuff.com/2010/08/how-to-create-lvm/

https://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/VG_admin.html

http://ubuntuforums.org/showthread.php?t=884556

Accessing ILOM on Sunfire X4540

Connect to the ILOM via serial port.

1. Open up a terminal via the port number (e.g. COM3). The correct serial port can be viewed under My Computer > Device Manager > Ports

Login to the web GUI via NET MGT
http://docs.oracle.com/cd/E24707_01/html/E24528/z40019501400145.html#scrolltoc

1. Connect an ethernet cable to NET MGT.
2. Set your internet settings on host to use the ILOM's IP address, netmask, and gateway.

VirtualBox for Fedora Linux

https://www.virtualbox.org/wiki/Downloads

# cd /etc/yum.repos.d/
# wget http://download.virtualbox.org/virtualbox/rpm/fedora/virtualbox.repo 

# yum install VirtualBox

Note: tested on Fedora 19. Use generic VirtualBox as keyword

because some versions of Virtual Box may not have support for our kernel.

Formatting a Hard Disk

For disk < 2TB, use fdisk
http://www.cyberciti.biz/faq/linux-disk-format/

For disk > 2TB, use parted 2.3 and above
http://www.cyberciti.biz/tips/fdisk-unable-to-create-partition-greater-2tb.html

Wipe a Hard Disk
http://www.cyberciti.biz/faq/linux-remove-all-partitions-data-empty-disk/

To test out the performance of the disk with dd
http://www.cyberciti.biz/faq/howto-linux-unix-test-disk-performance-with-dd-command/

To set up NFS
http://hpc-sw.blogspot.com/2014/09/setting-up-two-node-hpc-cluster.html

Mellanox Switch Systems

Switch System brochure

As at 4 August 2015.

Setting Up IP over IB

We have 1 dual port HCA card installed. Assuming port 2 (ib1) is up and port 1 (ib0) is down.

Step 1

In /etc/sysconfig/network-scripts, create:

$ cat ifcfg-ib1
DEVICE=ib1
#HWADDR=80:00:00:49:FE:80:00:00:00:00:00:00:00:02:C9:03:00:0F:D8:5E
TYPE=InfiniBand
UUID=48c746a0-5536-408f-ae8f-c36be67c66c7
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
IPADDR=192.168.11.1
NETMASK=255.255.255.0

Step 2

$ /etc/init.d/openibd restart
$ /etc/init.d/network restart

Step 3

Verify the connectivity:

$ ifconfig
# Check if ib1 is up and has IP assigned

$ ibdev2netdev

Set up another node but with different IP, e.g. 192.168.11.2. From this new node you can ping the older node by
$ ping 192.168.11.1

Vi Tricks

http://stackoverflow.com/questions/235839/indent-multiple-lines-quickly-in-vi

Installing Google Chrome on CentOS 7.0 / 6.x

Create /etc/yum.repos.d/google-chrome.repo

Check if latest version of Chrome available from Google's repository by
# yum info google-chrome-stable

# yum install google-chrome-stable
This will fail.

Sadly, the Google Chrome browser no longer supports the most famous commercial distribution RHEL 6.x and its free clones such as CentOS and Scientific Linux.

However, there's this script by Richard Lloyd that

automatically download and install latest Google Chrome browser by picking libraries from a more recent released distro and put those libraries in (/opt/google/chrome/lib) directory and then you can able to run Google Chrome on RHEL/CentOS 6.x versions.

# wget http://chrome.richardlloyd.org.uk/install_chrome.sh

# chmod u+x install_chrome.sh

# ./install_chrome.sh


References
http://www.tecmint.com/install-google-chrome-on-redhat-centos-fedora-linux/

Cloning Compute Nodes

Assuming:

master is in sda

target is in sdb

dd if=/dev/sda of=/dev/sdb bs=32M

comment out entries in /etc/udev/rules.d/70-persistent-net.rules

Change hostname and IP of the master into the desired one.

Comment out HWADDR and UUID. 

Move sda to the new compute node.

Move sdb to sda of the original compute node. 

Data Transfer using Tar

Most common way of doing data transfer is scp and rsync. However, data transfer with tar is also possible.

An example using tar with pbzip2.

pbzip2 is a parallel compression utility.

From /opt/intel on host machine to /opt on a remote server called fuji103.

tar -cpf - --use-compress-prog=/apps/pbzip2/116/pbzip2 /opt/intel | ssh fuji103 "(source ~kevins/.bashrc; cd /opt/; tar --use-compress-prog=/apps/pbzip2/1.1.12/pbzip2 -xpf -)"

From NFS directory called /usr/local/opt/intel to local directory /opt.

tar -cpf - --use-compress-prog=/apps/pbzip2/116/pbzip2 /usr/local/opt/intel | tar --use-compress-prog=/apps/pbzip2/1.1.12/pbzip2 -xpf -

How to Check the Status of a License Server

On a remote server requiring the license:

#See some help messages
$ lmstat -h

See everything
$ lmstat -a

On the license server:

#Grab the port number
$ cat /usr/local/lsf/conf/license.dat

You can try to telnet to that port by
$ telnet localhost port_number

On the remote server:

#This will check the status of the license server on the remote server
$ lmstat -a -c port_number@license_server_address

Troubleshooting LDAP

Healthy:

slapd should be running

If not:

Check /var/log/ldap.log

If database corrupted, use db_recover -h <path-to-database>

<path-to-database> can be /var/lib/ldap, can be found on /etc/openldap/slapd.conf

A sample extract from /etc/openldap/slapd.conf
# The database directory MUST exist prior to running slapd AND
# should only be accessible by the slapd and slap tools.
# Mode 700 recommended.
directory       /var/lib/ldap

Make sure to:
$ chown -R ldap /var/lib/ldap && chgrp -R ldap /var/lib/ldap

If it does not work, check the backup (for instance /var/lib/ldap/backup) against the current database directory

Installing TCL

8.6.3

http://sourceforge.net/projects/tcl/files/Tcl/8.6.3/

8.6.1

Binary download: http://downloads.activestate.com/ActiveTcl/releases/8.6.1.0/

$ mkdir -p tcl/tcl-8.6.1

Untar and run install.sh

Installation Options

     Installation Directory:  /scratch1/dsi/dsinibal/tcl/tcl-8.6.1
     Demos Directory:         /scratch1/dsi/dsinibal/tcl/tcl-8.6.1/demos
     Runtime Directory:       See Installation Directory

Post-Install Messages

Please do not forget to extend your PATH and MANPATH variables to
get access to the applications and manpages distributed with ActiveTcl.

For a csh or compatible perform
    setenv PATH "/scratch1/dsi/dsinibal/tcl/tcl-8.6.1/bin:$PATH"

For a sh or similar perform
    PATH="/scratch1/dsi/dsinibal/tcl/tcl-8.6.1/bin:$PATH"
    export PATH

Some shells (bash for example) allow
    export PATH="/scratch1/dsi/dsinibal/tcl/tcl-8.6.1/bin:$PATH"

Similar changes are required for MANPATH


  Note that ActiveTcl 8.6.1.0 is a trimmed down distribution
  providing only the most important packages. All packages
  not found in the distribution can be installed by using
  the teacup client to the TEApot Package Management however.

  Further note that the documentation was not trimmed, and
  contains the documentation of all packages, even those not
  installed by the distribution.

Installing GCC 4.9.1 from source on Fuji

To be installed: GMP 6.0.0, MPFR 3.1.2, MPC 1.0.2, GCC 4.9.1

The source codes can be obtained from one of the mirrors.
Example:
ftp://gcc.gnu.org/pub/gcc/infrastructure/

GMP is needed by MPFR, which are needed by MPC, which are needed by GCC.

GMP

$ tar -xzvf gmp-6.0.0a.tar.bz2

$ rsync -avr /apps/GNU/GMP/6.0.0/ gmp-6.0.0/*

$ cd /apps/GNU/GMP/6.0.0

$ ./configure --disable-shared --enable-static --prefix=/apps/GNU/GMP/6.0.0

$ make && make check && make install

MPFR

$ tar -xzvf mpfr-3.1.2.tar.gz

$ rsync -avr /apps/GNU/MPFR/3.1.2-new/ mpfr-3.1.2/*

$ cd /apps/GNU/MPFR/3.1.2-new

$ ./configure --disable-shared --enable-static --prefix=/apps/GNU/MPFR/3.1.2-new --with-gmp=/apps/GNU/GMP/6.0.0 

$ make && make check && make install

MPC

$ tar -xzvf mpc-1.0.2.tar.gz

$ rsync -avr /apps/GNU/MPC/1.0.2/ mpc-1.0.2/*

$ cd /apps/GNU/MPC/1.0.2

$ ./configure --disable-shared --enable-static --prefix=/apps/GNU/MPC/1.0.2 --with-gmp=/apps/GNU/GMP/6.0.0 --with-mpfr=/apps/GNU/MPFR/3.1.2-new

$ make && make check && make install

GCC

$ tar -xzvf gcc-4.9.1.tar.gz

$ cd /apps/GNU/GCC/4.9.1

$ <path-to-gcc-source>/gcc-4.9.1/configure --with-gmp=/apps/GNU/GMP/6.0.0 --with-mpfr=/apps/GNU/MPFR/3.1.2-new --with-mpc=/apps/GNU/MPC/1.0.2 --disable-multilib

$ make #This will take a long time

$ make install

After successful build, there is one important message:

Libraries have been installed in:
   /apps/GNU/GCC/4.9.1/lib/../lib64

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.

OpenFOAM 2.3.0 in Fuji

System: Fuji (Upgraded - CentOS 6.5)
Note: I used OpenMPI here for this test installation. Default MPI in Fuji is Intel MPI.

Download:
OpenFOAM-2.3.0.tgz
ThirdParty-2.3.0.tgz

Setting up GCC 4.9.1

GCC 4.9.1 is available from /apps/GNU/GCC/4.9.1

$ export PATH=/apps/GNU/GCC/4.9.1/bin:$PATH

$ export LD_LIBRARY_PATH=/apps/GNU/GCC/4.9.1/lib64:/apps/GNU/GCC/4.9.1/lib:/apps/GNU/MPC/1.0.2/lib:/apps/GNU/GMP/6.0.0/lib:/apps/GNU/MPFR/3.1.2/lib:$LD_LIBRARY_PATH

Setting up OpenFOAM Installation

$ mkdir ~/scratch/OpenFOAM

Download the OpenFOAM and ThirdParty into this directory.

$ tar -xzvf OpenFOAM-2.3.0.tgz
$ tar -xzvf ThirdParty-2.3.0.tgz

$ export FOAM_INST_DIR=~/scratch/OpenFOAM
$ foamDotFile=$FOAM_INST_DIR/OpenFOAM-2.3.0/etc/bashrc
$ [ -f $foamDotFile ] && . $foamDotFile


$ cd $FOAM_INST_DIR
$ mkdir obj
$ cd obj
$ ../OpenFOAM-2.3.0/Allwmake

Testing

Notice that all the executables, e.g. icoFoam, are installed on $FOAM_INST_DIR/bin.


In the remote machine, set the .bashrc to include:

export FOAM_INST_DIR=$HOME/scratch/OpenFOAM
source $FOAM_INST_DIR/OpenFOAM-2.3.0/etc/bashrc


To run a test parallel OpenFOAM on 2 nodes:

$ mkdir -p $FOAM_RUN

$ cp -r $FOAM_TUTORIALS $FOAM_RUN

$ cd $FOAM_RUN/tutorials/incompressible/icoFoam

$ cp -r cavity cavityParallel

Copy $WM_PROJECT_DIR/applications/utilities/parallelProcessing/decomposePar/decomposeParDict to cavityParallel/system

Edit the decomposeParDict:
numberOfSubdomains  2;
method simple;

$ cd cavityParallel

$ blockMesh

$ cd ..

$ decomposePar -case cavityParallel

$ cd cavityParallel

$ echo -e 'fuji381\nfuji382' > hosts

$ mpirun -f hosts -np 2 icoFoam -parallel

Installing OFED on Linux (CentOS 6.5)

As the title suggests, I will show how to install OFED stack on CentOS 6.5.

Prerequisites

kernel-devel
rpm-build
libtool
gcc-c++
bison
flex
glib2-devel
glib2
tcl-devel
zlib-devel

Tips:

• To prevent build error, make sure your gcc version is your kernel's latest.
• It's recommended to use the latest kernel from the repo.

Download OFED software from https://www.openfabrics.org/index.php

Extract and run install.pl (--help to see options)

After installation, do a reboot

Some points:

• The openib daemon controls which module to load at startup.
• ibstat can be used to check if card (mlx - mellanox, qib - qlogic) is up
• ibhosts and iblinkinfo may be useful
• ibping the other nodes for sanity checking. See http://hpc-sw.blogspot.sg/2014/09/setting-up-two-node-hpc-cluster-with.html

Typically, locked memory limit has to be set to unlimited to be able to run HPC MPI jobs across nodes. 

Add the following to /etc/security/limits.conf:

* soft memlock unlimited
* hard memlock unlimited

Exit the shell and you should have:
$ ulimit -l
unlimited

Intel MPI How To Use and Debug

Running /bin/hostname

MPIRUN directory:
/opt/intel/impi/<version>/intel64/bin

Source mpivars.sh

Create a machinefile:
$ cat mach.txt
node1
node2

Test run:
$ mpirun -r ssh -f mach.txt -ppn 1 -np 2 ./bin/hostname

mpirun is a utility which runs mpdboot after that mpiexec. So, options for mpdboot comes first and after that options for mpiexec. '-machinefile' is an option for mpiexec.
With mpirun, there is actually no need to run mpdboot (needed only for mpiexec) nor creating mpd.hosts.

If instead you would like to use mpiexec, you would have to do the following.

Create mpd.hosts on your working directory.

Example is
$ cat mpd.hosts
node1
node2

Start mpdring:
$ mpdboot

Try another one: cpi.c
$ mpiicpc mpi.c
$ mpirun -f mach.txt -ppn 4 -np 8 ./a.out

Debugging

Note: Sometimes iptables might prevent mpi across nodes. You might want to flush or edit iptables.

Debugging:

#Pass DEBUG environment variables
export I_MPI_DEBUG=5

#Check mpd is up
$ mpdtrace

#To specifically use IB HCA port 2 instead of default port 1
export I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-2

Note: DAPL versions of the nodes must match. Older versions of Intel MPI do not support DAPL v2.0. When installing the OS, make sure the necessary Infiniband drivers (e.g. DAPL 1.2 if using old Intel MPI) are installed.

Configuring Internet Connection For Compute Nodes Of A Cluster

Idea: In a cluster, usually only the head node has outgoing internet connection. This post details how to set up the compute nodes to have outgoing internet connection using the head node as the router.

Assuming enp129s0f0 is for internal network and enp129s0f1 for external network.

1. Tell kernel to allow ip forwarding:

On the head node 

$ echo 1 > /proc/sys/net/ipv4/ip_forward

2. Configure IPTABLES to forward packets from internal network.

On the head node
$ sudo iptables -t nat -A POSTROUTING -o enp129s0f1 -j MASQUERADE
$ sudo iptables -A FORWARD -i enp129s0f0 -o enp129s0f1 -j ACCEPT
$ sudo iptables -A FORWARD -i enp129s0f1 -o enp129s0f0 -m state --state RELATED,ESTABLISHED -j ACCEPT
$ sudo iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Debugging SSH without-password

Sometimes, we want to set up passwordless login across nodes. In this post, I will use the root user as an example.

Common Practice

Easiest way:
$ ssh-copy-id remotehostname

Sometimes, we may encounter
/usr/bin/ssh-copy-id: ERROR: No identities found

In that case, do:
$ ssh-copy-id -i ~/.ssh/id_rsa.pub remotehostname

If known_hosts has offending key
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
   .
   .
   .
Offending key in /root/.ssh/known_hosts: 6
   .
   .