Installing MPB on High Performance Computing Clusters

MPB Main Page: http://ab-initio.mit.edu/wiki/index.php/MIT_Photonic_Bands

In /usr/local/lib & /usr/local/include, check if you have the libraries and header files, respectively, for

• BLAS (libblas.a)
• LAPACK (liblapack.a)

Installing libctl

Reference: http://ab-initio.mit.edu/wiki/index.php/Libctl

Install FFTW

Reference: http://www.fftw.org/fftw3_doc/Installation-and-Customization.html

Important: you will also need to install the MPI FFTW libraries (this just means including --enable-mpi in the FFTW configure flags)

On SGI (Silicon Graphics) machine

export LDFLAGS="-L$HOME/scratch/fftw-3.3.4/lib -L/apps/HDF5/1.8.14/share/lib -L//apps/MPB/LIBCTL/3.2.2/lib"

export CPPFLAGS="-I$HOME/scratch/fftw-3.3.4/include -I/apps/HDF5/1.8.14/include -I/apps/MPB/LIBCTL/3.2.2/include"

export WM_CFLAGS="$WM_CFLAGS -DMPI_NO_CPPBIND -DSGIMPI"

gcc -lmpi

./configure --prefix=$HOME/scratch/mpb/ --with-mpi - --with-libctl=$HOME/scratch/libctl/libctl-3.2.2/share/libctl

On Fujitsu machine

export LDFLAGS="-L/apps/FFTW/3.3.4-gcc-4.8.2/lib -L"
export CPPFLAGS="-I/apps/FFTW/3.3.4-gcc-4.8.2/include -I"

Creating RAID 6 in Linux

Goal: 4x RAID 6 arrays
Each array will consist of 10 disks + 2 spares.

For the first array:

Use fdisk to partition the HDs
Active: /dev/sd[b-k]
Spare: /dev/sd[l-m]

Examine the disks
mdadm -E /dev/sd[b-k]1

Create RAID 6 array
mdadm --create /dev/md0 --level=6 --raid-devices=10 /dev/sd[b-k]1

Create 2 spare disks
mdadm --add /dev/md0 /dev/sdl1 /dev/sdm1

Sanity check
cat /proc/mdstat
mdadm -E /dev/sd[b-m]1
mdadm --detail /dev/md0

Do the same for the remaining 3 arrays. Remember to save the mdadm config by

mdadm --detail --scan --verbose >> /etc/mdadm.conf

LVM

vgcreate vg1 /dev/md0 /dev/md1 /dev/md2

lvcreate -l 100%FREE -n lv1 vg1

mkfs.xfs /dev/vg1/lv1

mount -t xfs /dev/vg1/lv1 /mnt/test


In case of one array fails.

lvremove /dev/vg1/lv1

vgreduce vg1 --removemissing

#Remove the offending array, e.g /dev/md0
mdadm --stop /dev/md0
mdadm --zero-superblock /dev/sd[b-m]1

#Recreate the RAID array.

#Then add to vg1
vgextend vg1 /dev/md0

#Recreate the volume group and mount.
-------------------
References

http://www.tecmint.com/create-raid-6-in-linux/

http://www.thegeekstuff.com/2010/08/how-to-create-lvm/

https://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/VG_admin.html

http://ubuntuforums.org/showthread.php?t=884556

Accessing ILOM on Sunfire X4540

Connect to the ILOM via serial port.

1. Open up a terminal via the port number (e.g. COM3). The correct serial port can be viewed under My Computer > Device Manager > Ports

Login to the web GUI via NET MGT
http://docs.oracle.com/cd/E24707_01/html/E24528/z40019501400145.html#scrolltoc

1. Connect an ethernet cable to NET MGT.
2. Set your internet settings on host to use the ILOM's IP address, netmask, and gateway.

VirtualBox for Fedora Linux

https://www.virtualbox.org/wiki/Downloads

# cd /etc/yum.repos.d/
# wget http://download.virtualbox.org/virtualbox/rpm/fedora/virtualbox.repo 

# yum install VirtualBox

Note: tested on Fedora 19. Use generic VirtualBox as keyword

because some versions of Virtual Box may not have support for our kernel.

Formatting a Hard Disk

For disk < 2TB, use fdisk
http://www.cyberciti.biz/faq/linux-disk-format/

For disk > 2TB, use parted 2.3 and above
http://www.cyberciti.biz/tips/fdisk-unable-to-create-partition-greater-2tb.html

Wipe a Hard Disk
http://www.cyberciti.biz/faq/linux-remove-all-partitions-data-empty-disk/

To test out the performance of the disk with dd
http://www.cyberciti.biz/faq/howto-linux-unix-test-disk-performance-with-dd-command/

To set up NFS
http://hpc-sw.blogspot.com/2014/09/setting-up-two-node-hpc-cluster.html

Mellanox Switch Systems

Switch System brochure

As at 4 August 2015.

Setting Up IP over IB

We have 1 dual port HCA card installed. Assuming port 2 (ib1) is up and port 1 (ib0) is down.

Step 1

In /etc/sysconfig/network-scripts, create:

$ cat ifcfg-ib1
DEVICE=ib1
#HWADDR=80:00:00:49:FE:80:00:00:00:00:00:00:00:02:C9:03:00:0F:D8:5E
TYPE=InfiniBand
UUID=48c746a0-5536-408f-ae8f-c36be67c66c7
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
IPADDR=192.168.11.1
NETMASK=255.255.255.0

Step 2

$ /etc/init.d/openibd restart
$ /etc/init.d/network restart

Step 3

Verify the connectivity:

$ ifconfig
# Check if ib1 is up and has IP assigned

$ ibdev2netdev

Set up another node but with different IP, e.g. 192.168.11.2. From this new node you can ping the older node by
$ ping 192.168.11.1