Installing OFED on Linux (CentOS 6.5)

As the title suggests, I will show how to install OFED stack on CentOS 6.5.

Prerequisites

kernel-devel
rpm-build
libtool
gcc-c++
bison
flex
glib2-devel
glib2
tcl-devel
zlib-devel

Tips:

• To prevent build error, make sure your gcc version is your kernel's latest.
• It's recommended to use the latest kernel from the repo.

Download OFED software from https://www.openfabrics.org/index.php

Extract and run install.pl (--help to see options)

After installation, do a reboot

Some points:

• The openib daemon controls which module to load at startup.
• ibstat can be used to check if card (mlx - mellanox, qib - qlogic) is up
• ibhosts and iblinkinfo may be useful
• ibping the other nodes for sanity checking. See http://hpc-sw.blogspot.sg/2014/09/setting-up-two-node-hpc-cluster-with.html

Typically, locked memory limit has to be set to unlimited to be able to run HPC MPI jobs across nodes. 

Add the following to /etc/security/limits.conf:

* soft memlock unlimited
* hard memlock unlimited

Exit the shell and you should have:
$ ulimit -l
unlimited