Cloning Compute Nodes

Assuming:

master is in sda

target is in sdb

dd if=/dev/sda of=/dev/sdb bs=32M

comment out entries in /etc/udev/rules.d/70-persistent-net.rules

Change hostname and IP of the master into the desired one.

Comment out HWADDR and UUID. 

Move sda to the new compute node.

Move sdb to sda of the original compute node. 

Data Transfer using Tar

Most common way of doing data transfer is scp and rsync. However, data transfer with tar is also possible.

An example using tar with pbzip2.

pbzip2 is a parallel compression utility.

From /opt/intel on host machine to /opt on a remote server called fuji103.

tar -cpf - --use-compress-prog=/apps/pbzip2/116/pbzip2 /opt/intel | ssh fuji103 "(source ~kevins/.bashrc; cd /opt/; tar --use-compress-prog=/apps/pbzip2/1.1.12/pbzip2 -xpf -)"

From NFS directory called /usr/local/opt/intel to local directory /opt.

tar -cpf - --use-compress-prog=/apps/pbzip2/116/pbzip2 /usr/local/opt/intel | tar --use-compress-prog=/apps/pbzip2/1.1.12/pbzip2 -xpf -

How to Check the Status of a License Server

On a remote server requiring the license:

#See some help messages
$ lmstat -h

See everything
$ lmstat -a

On the license server:

#Grab the port number
$ cat /usr/local/lsf/conf/license.dat

You can try to telnet to that port by
$ telnet localhost port_number

On the remote server:

#This will check the status of the license server on the remote server
$ lmstat -a -c port_number@license_server_address

Troubleshooting LDAP

Healthy:

slapd should be running

If not:

Check /var/log/ldap.log

If database corrupted, use db_recover -h <path-to-database>

<path-to-database> can be /var/lib/ldap, can be found on /etc/openldap/slapd.conf

A sample extract from /etc/openldap/slapd.conf
# The database directory MUST exist prior to running slapd AND
# should only be accessible by the slapd and slap tools.
# Mode 700 recommended.
directory       /var/lib/ldap

Make sure to:
$ chown -R ldap /var/lib/ldap && chgrp -R ldap /var/lib/ldap

If it does not work, check the backup (for instance /var/lib/ldap/backup) against the current database directory

Installing TCL

8.6.3

http://sourceforge.net/projects/tcl/files/Tcl/8.6.3/

8.6.1

Binary download: http://downloads.activestate.com/ActiveTcl/releases/8.6.1.0/

$ mkdir -p tcl/tcl-8.6.1

Untar and run install.sh

Installation Options

     Installation Directory:  /scratch1/dsi/dsinibal/tcl/tcl-8.6.1
     Demos Directory:         /scratch1/dsi/dsinibal/tcl/tcl-8.6.1/demos
     Runtime Directory:       See Installation Directory

Post-Install Messages

Please do not forget to extend your PATH and MANPATH variables to
get access to the applications and manpages distributed with ActiveTcl.

For a csh or compatible perform
    setenv PATH "/scratch1/dsi/dsinibal/tcl/tcl-8.6.1/bin:$PATH"

For a sh or similar perform
    PATH="/scratch1/dsi/dsinibal/tcl/tcl-8.6.1/bin:$PATH"
    export PATH

Some shells (bash for example) allow
    export PATH="/scratch1/dsi/dsinibal/tcl/tcl-8.6.1/bin:$PATH"

Similar changes are required for MANPATH


  Note that ActiveTcl 8.6.1.0 is a trimmed down distribution
  providing only the most important packages. All packages
  not found in the distribution can be installed by using
  the teacup client to the TEApot Package Management however.

  Further note that the documentation was not trimmed, and
  contains the documentation of all packages, even those not
  installed by the distribution.

Installing GCC 4.9.1 from source on Fuji

To be installed: GMP 6.0.0, MPFR 3.1.2, MPC 1.0.2, GCC 4.9.1

The source codes can be obtained from one of the mirrors.
Example:
ftp://gcc.gnu.org/pub/gcc/infrastructure/

GMP is needed by MPFR, which are needed by MPC, which are needed by GCC.

GMP

$ tar -xzvf gmp-6.0.0a.tar.bz2

$ rsync -avr /apps/GNU/GMP/6.0.0/ gmp-6.0.0/*

$ cd /apps/GNU/GMP/6.0.0

$ ./configure --disable-shared --enable-static --prefix=/apps/GNU/GMP/6.0.0

$ make && make check && make install

MPFR

$ tar -xzvf mpfr-3.1.2.tar.gz

$ rsync -avr /apps/GNU/MPFR/3.1.2-new/ mpfr-3.1.2/*

$ cd /apps/GNU/MPFR/3.1.2-new

$ ./configure --disable-shared --enable-static --prefix=/apps/GNU/MPFR/3.1.2-new --with-gmp=/apps/GNU/GMP/6.0.0 

$ make && make check && make install

MPC

$ tar -xzvf mpc-1.0.2.tar.gz

$ rsync -avr /apps/GNU/MPC/1.0.2/ mpc-1.0.2/*

$ cd /apps/GNU/MPC/1.0.2

$ ./configure --disable-shared --enable-static --prefix=/apps/GNU/MPC/1.0.2 --with-gmp=/apps/GNU/GMP/6.0.0 --with-mpfr=/apps/GNU/MPFR/3.1.2-new

$ make && make check && make install

GCC

$ tar -xzvf gcc-4.9.1.tar.gz

$ cd /apps/GNU/GCC/4.9.1

$ <path-to-gcc-source>/gcc-4.9.1/configure --with-gmp=/apps/GNU/GMP/6.0.0 --with-mpfr=/apps/GNU/MPFR/3.1.2-new --with-mpc=/apps/GNU/MPC/1.0.2 --disable-multilib

$ make #This will take a long time

$ make install

After successful build, there is one important message:

Libraries have been installed in:
   /apps/GNU/GCC/4.9.1/lib/../lib64

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.

OpenFOAM 2.3.0 in Fuji

System: Fuji (Upgraded - CentOS 6.5)
Note: I used OpenMPI here for this test installation. Default MPI in Fuji is Intel MPI.

Download:
OpenFOAM-2.3.0.tgz
ThirdParty-2.3.0.tgz

Setting up GCC 4.9.1

GCC 4.9.1 is available from /apps/GNU/GCC/4.9.1

$ export PATH=/apps/GNU/GCC/4.9.1/bin:$PATH

$ export LD_LIBRARY_PATH=/apps/GNU/GCC/4.9.1/lib64:/apps/GNU/GCC/4.9.1/lib:/apps/GNU/MPC/1.0.2/lib:/apps/GNU/GMP/6.0.0/lib:/apps/GNU/MPFR/3.1.2/lib:$LD_LIBRARY_PATH

Setting up OpenFOAM Installation

$ mkdir ~/scratch/OpenFOAM

Download the OpenFOAM and ThirdParty into this directory.

$ tar -xzvf OpenFOAM-2.3.0.tgz
$ tar -xzvf ThirdParty-2.3.0.tgz

$ export FOAM_INST_DIR=~/scratch/OpenFOAM
$ foamDotFile=$FOAM_INST_DIR/OpenFOAM-2.3.0/etc/bashrc
$ [ -f $foamDotFile ] && . $foamDotFile


$ cd $FOAM_INST_DIR
$ mkdir obj
$ cd obj
$ ../OpenFOAM-2.3.0/Allwmake

Testing

Notice that all the executables, e.g. icoFoam, are installed on $FOAM_INST_DIR/bin.


In the remote machine, set the .bashrc to include:

export FOAM_INST_DIR=$HOME/scratch/OpenFOAM
source $FOAM_INST_DIR/OpenFOAM-2.3.0/etc/bashrc


To run a test parallel OpenFOAM on 2 nodes:

$ mkdir -p $FOAM_RUN

$ cp -r $FOAM_TUTORIALS $FOAM_RUN

$ cd $FOAM_RUN/tutorials/incompressible/icoFoam

$ cp -r cavity cavityParallel

Copy $WM_PROJECT_DIR/applications/utilities/parallelProcessing/decomposePar/decomposeParDict to cavityParallel/system

Edit the decomposeParDict:
numberOfSubdomains  2;
method simple;

$ cd cavityParallel

$ blockMesh

$ cd ..

$ decomposePar -case cavityParallel

$ cd cavityParallel

$ echo -e 'fuji381\nfuji382' > hosts

$ mpirun -f hosts -np 2 icoFoam -parallel