Configuring Internet Connection For Compute Nodes Of A Cluster

Idea: In a cluster, usually only the head node has outgoing internet connection. This post details how to set up the compute nodes to have outgoing internet connection using the head node as the router.

Assuming enp129s0f0 is for internal network and enp129s0f1 for external network.

1. Tell kernel to allow ip forwarding:

On the head node 

$ echo 1 > /proc/sys/net/ipv4/ip_forward

2. Configure IPTABLES to forward packets from internal network.

On the head node
$ sudo iptables -t nat -A POSTROUTING -o enp129s0f1 -j MASQUERADE
$ sudo iptables -A FORWARD -i enp129s0f0 -o enp129s0f1 -j ACCEPT
$ sudo iptables -A FORWARD -i enp129s0f1 -o enp129s0f0 -m state --state RELATED,ESTABLISHED -j ACCEPT
$ sudo iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

3. Ensure interface settings are correct.

On head node:

$ cat /etc/sysconfig/network-scripts/ifcfg-enp129s0f0
DEVICE=enp129s0f0
BOOTPROTO=none
HWADDR=00:26:2D:00:0A:26 #Unique, no need to change
ONBOOT=yes
DHCP_HOSTNAME=fuji
TYPE=Ethernet
IPADDR=10.10.8.243
NETMASK=255.255.0.0
GATEWAY=10.10.8.254 #not necessary, it's here for other purposes
USERCTL=no
IPV6INIT=no
PEERDNS=yes

$ cat /etc/sysconfig/network-scripts/ifcfg-enp129s0f1
DEVICE=eth1
BOOTPROTO=none
HWADDR=00:1B:21:52:9A:BD #Unique, no need to change
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
IPV6INIT=no
PEERDNS=yes
IPADDR=xxx.xx.xxx.xx
NETWORK=xxx.xx.xxx.0
NETMASK=255.255.255.0
BROADCAST=xxx.xx.xxx.255
GATEWAY=xxx.xx.xxx.1

On compute nodes, e.g. node1:

$ cat /etc/sysconfig/network-scripts/ifcfg-enp129s0f0
DEVICE=enp129s0f0
HWADDR=60:EB:69:BA:DA:10 #Unique, no need to change
TYPE=Ethernet
UUID=720917c3-1e2b-42ee-94e7-e7cd47cadcb6 #Unique, no need to change
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
IPADDR=10.10.7.1
NETMASK=255.255.0.0

$ cat /etc/resolv.conf
nameserver 8.8.8.8

$ cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=node1
GATEWAY=10.10.8.243 #Address of headnode

4. To make the settings persistent over reboots.

From: http://www.revsys.com/writings/quicktips/nat.html
You will need to edit /etc/sysctl.conf and change the line that says net.ipv4.ip_forward = 0 to net.ipv4.ip_forward = 1. Notice how this is similar to step number one? This essentially tells your kernel to do step one on boot.
Ok last step for Fedora/RHEL users. In order for your system to save the iptables rules we setup in step two you have to configure iptables correctly. You will need to edit /etc/sysconfig/iptables-config and make sure IPTABLES_MODULES_UNLOAD, IPTABLES_SAVE_ON_STOP, and IPTABLES_SAVE_ON_RESTART are all set to 'yes'

For non-Fedora/RHEL users you can simply setup an init script for this or simply append these commands to the existing rc.local script so they are executed on boot. Or if you want to get even more fancy, you can use the commands iptables-save and iptables-restore to save/restore the current state of your iptables rules.