Configuring Internet Connection For Compute Nodes Of A Cluster

Idea: In a cluster, usually only the head node has outgoing internet connection. This post details how to set up the compute nodes to have outgoing internet connection using the head node as the router.

Assuming enp129s0f0 is for internal network and enp129s0f1 for external network.

1. Tell kernel to allow ip forwarding:

On the head node 

$ echo 1 > /proc/sys/net/ipv4/ip_forward

2. Configure IPTABLES to forward packets from internal network.

On the head node
$ sudo iptables -t nat -A POSTROUTING -o enp129s0f1 -j MASQUERADE
$ sudo iptables -A FORWARD -i enp129s0f0 -o enp129s0f1 -j ACCEPT
$ sudo iptables -A FORWARD -i enp129s0f1 -o enp129s0f0 -m state --state RELATED,ESTABLISHED -j ACCEPT
$ sudo iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Configuring a Compute Node (of a Supercomputer)

This post details how a compute node is set up after an upgrade (e.g. OS update).
This instruction is based on the real setup of one HPC system that I administered.

Migrate User Accounts

Write a script to transfer user accounts. This step is for creating user home directories cleanly (optional).
Require: /etc/shadow and /etc/passwd from login node.

Next, copy over the /etc/passwd, /etc/shadow, /etc/group, and /etc/hosts from master node. Remember to make copies of the existing files on the compute node.

Transfer user home directory to /home01
$ cd /
$ ln -s home home01

Set up internet

$ echo 'GATEWAY=10.10.8.243' >> /etc/sysconfig/network

$ echo -e 'nameserver 10.10.8.236\nnameserver 202.83.248.3\nnameserver 123.136.66.68' >> /etc/resolv.conf

$ service network restart

Flush iptables and make a copy of the existing one. Comment up the rules in /etc/sysconfig/iptables.

Setting up a two-node HPC cluster with InfiniBand

In this post, I would share how to build a two-node cluster with InfiniBand (IB) interconnection.

I have installed CentOS 7.0 (server with GUI version) with IB and iWARP support.

To install the necessary software

The following are to be done on every node, unless stated otherwise.

$ yum groupinstall "Infiniband Support"
$ yum install infiniband-diags perftest qperf opensm

OpenSM is the subnet manager.
 
Extra steps (optional):

Edit /etc/default/opensm such that
$ cat /etc/default/opensm
PORTS="0x00117500007005aa 0x0011750000700c2a"

The port GUID can be obtained by doing
$ ibstat -p

Activate the services:
$ chkconfig rdma on
$ chkconfig opensm on #only on master node

$ service rdma start
$ service opensm start #only on master node
$ shutdown -r now

After services are started / reboot, hopefully ibstat will show State "Active" and Physical State "LinkUp".

To check network connectivity

Display all switches
$ ibswitches

Display all hosts visible in the network
$ ibhosts

Reports link info
$ iblinkinfo

Testing with ibping
$ ibping -S #on one server
$ ibping -G 0x0011750000700c2a #on another server

Replace the port GUID above with yours.

Setting up a two-node ethernet cluster

In this post I would like to share on how to set up a two-node cluster.

First: Configuring the ethernet connection

For every node, we have to configure the network script for the ethernet card, /etc/hosts, /etc/hostname, /etc/resolv.conf, /etc/hosts.allow (and /etc/hosts.deny), and the iptables. The goal is the following:

• Assign private static IPs to every node.
• Assign hostnames to every node.
• Assign a virtual gateway and nameserver
• Allowing communication between the nodes through iptables (if applicable) and hosts.allow.

Let's say we have two nodes and we would like to name them node1 (head node) and node2.

Note: On node1 (chosen as the head node), enp3s0f0 is connected to the internet. So I used enp3s0f1 for the local network.