Creating RAID 6 in Linux

Goal: 4x RAID 6 arrays
Each array will consist of 10 disks + 2 spares.

For the first array:

Use fdisk to partition the HDs
Active: /dev/sd[b-k]
Spare: /dev/sd[l-m]

Examine the disks
mdadm -E /dev/sd[b-k]1

Create RAID 6 array
mdadm --create /dev/md0 --level=6 --raid-devices=10 /dev/sd[b-k]1

Create 2 spare disks
mdadm --add /dev/md0 /dev/sdl1 /dev/sdm1

Sanity check
cat /proc/mdstat
mdadm -E /dev/sd[b-m]1
mdadm --detail /dev/md0

Do the same for the remaining 3 arrays. Remember to save the mdadm config by

mdadm --detail --scan --verbose >> /etc/mdadm.conf

LVM

vgcreate vg1 /dev/md0 /dev/md1 /dev/md2

lvcreate -l 100%FREE -n lv1 vg1

mkfs.xfs /dev/vg1/lv1

mount -t xfs /dev/vg1/lv1 /mnt/test


In case of one array fails.

lvremove /dev/vg1/lv1

vgreduce vg1 --removemissing

#Remove the offending array, e.g /dev/md0
mdadm --stop /dev/md0
mdadm --zero-superblock /dev/sd[b-m]1

#Recreate the RAID array.

#Then add to vg1
vgextend vg1 /dev/md0

#Recreate the volume group and mount.
-------------------
References

http://www.tecmint.com/create-raid-6-in-linux/

http://www.thegeekstuff.com/2010/08/how-to-create-lvm/

https://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/VG_admin.html

http://ubuntuforums.org/showthread.php?t=884556

Accessing ILOM on Sunfire X4540

Connect to the ILOM via serial port.

1. Open up a terminal via the port number (e.g. COM3). The correct serial port can be viewed under My Computer > Device Manager > Ports

Login to the web GUI via NET MGT
http://docs.oracle.com/cd/E24707_01/html/E24528/z40019501400145.html#scrolltoc

1. Connect an ethernet cable to NET MGT.
2. Set your internet settings on host to use the ILOM's IP address, netmask, and gateway.

VirtualBox for Fedora Linux

https://www.virtualbox.org/wiki/Downloads

# cd /etc/yum.repos.d/
# wget http://download.virtualbox.org/virtualbox/rpm/fedora/virtualbox.repo 

# yum install VirtualBox

Note: tested on Fedora 19. Use generic VirtualBox as keyword

because some versions of Virtual Box may not have support for our kernel.

Formatting a Hard Disk

For disk < 2TB, use fdisk
http://www.cyberciti.biz/faq/linux-disk-format/

For disk > 2TB, use parted 2.3 and above
http://www.cyberciti.biz/tips/fdisk-unable-to-create-partition-greater-2tb.html

Wipe a Hard Disk
http://www.cyberciti.biz/faq/linux-remove-all-partitions-data-empty-disk/

To test out the performance of the disk with dd
http://www.cyberciti.biz/faq/howto-linux-unix-test-disk-performance-with-dd-command/

To set up NFS
http://hpc-sw.blogspot.com/2014/09/setting-up-two-node-hpc-cluster.html

Mellanox Switch Systems

Switch System brochure

As at 4 August 2015.

Setting Up IP over IB

We have 1 dual port HCA card installed. Assuming port 2 (ib1) is up and port 1 (ib0) is down.

Step 1

In /etc/sysconfig/network-scripts, create:

$ cat ifcfg-ib1
DEVICE=ib1
#HWADDR=80:00:00:49:FE:80:00:00:00:00:00:00:00:02:C9:03:00:0F:D8:5E
TYPE=InfiniBand
UUID=48c746a0-5536-408f-ae8f-c36be67c66c7
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
IPADDR=192.168.11.1
NETMASK=255.255.255.0

Step 2

$ /etc/init.d/openibd restart
$ /etc/init.d/network restart

Step 3

Verify the connectivity:

$ ifconfig
# Check if ib1 is up and has IP assigned

$ ibdev2netdev

Set up another node but with different IP, e.g. 192.168.11.2. From this new node you can ping the older node by
$ ping 192.168.11.1

Vi Tricks

http://stackoverflow.com/questions/235839/indent-multiple-lines-quickly-in-vi

Installing Google Chrome on CentOS 7.0 / 6.x

Create /etc/yum.repos.d/google-chrome.repo

Check if latest version of Chrome available from Google's repository by
# yum info google-chrome-stable

# yum install google-chrome-stable
This will fail.

Sadly, the Google Chrome browser no longer supports the most famous commercial distribution RHEL 6.x and its free clones such as CentOS and Scientific Linux.

However, there's this script by Richard Lloyd that

automatically download and install latest Google Chrome browser by picking libraries from a more recent released distro and put those libraries in (/opt/google/chrome/lib) directory and then you can able to run Google Chrome on RHEL/CentOS 6.x versions.

# wget http://chrome.richardlloyd.org.uk/install_chrome.sh

# chmod u+x install_chrome.sh

# ./install_chrome.sh


References
http://www.tecmint.com/install-google-chrome-on-redhat-centos-fedora-linux/

Cloning Compute Nodes

Assuming:

master is in sda

target is in sdb

dd if=/dev/sda of=/dev/sdb bs=32M

comment out entries in /etc/udev/rules.d/70-persistent-net.rules

Change hostname and IP of the master into the desired one.

Comment out HWADDR and UUID. 

Move sda to the new compute node.

Move sdb to sda of the original compute node. 

Data Transfer using Tar

Most common way of doing data transfer is scp and rsync. However, data transfer with tar is also possible.

An example using tar with pbzip2.

pbzip2 is a parallel compression utility.

From /opt/intel on host machine to /opt on a remote server called fuji103.

tar -cpf - --use-compress-prog=/apps/pbzip2/116/pbzip2 /opt/intel | ssh fuji103 "(source ~kevins/.bashrc; cd /opt/; tar --use-compress-prog=/apps/pbzip2/1.1.12/pbzip2 -xpf -)"

From NFS directory called /usr/local/opt/intel to local directory /opt.

tar -cpf - --use-compress-prog=/apps/pbzip2/116/pbzip2 /usr/local/opt/intel | tar --use-compress-prog=/apps/pbzip2/1.1.12/pbzip2 -xpf -

How to Check the Status of a License Server

On a remote server requiring the license:

#See some help messages
$ lmstat -h

See everything
$ lmstat -a

On the license server:

#Grab the port number
$ cat /usr/local/lsf/conf/license.dat

You can try to telnet to that port by
$ telnet localhost port_number

On the remote server:

#This will check the status of the license server on the remote server
$ lmstat -a -c port_number@license_server_address