Deploying Xeon Phi equipped nodes is a 2 steps process:
- the software stack needs to be deployed on the compute nodes
- the MIC(s) need to be configured
The first part is easy as the software stack is a set of rpms. The procedure is as follow:
- copy the rpms in the appropriate contrib directory (by default after installation: /depot/contrib/1000
- refresh the repository (repoman)
- add the rpms to the list of rpms to be deployed for a specific nodegroup (ngedit)
- synchronize / redeploy
For the second part, we saw previously how to create a bridge interface on the compute nodes and make sure its configuration will resist synchronization / reinstall. We just need to extend our script so it includes the MIC configuration part.
We assume here there is only one Xeon Phi per compute node, so the script needs to be extended if the compute nodes are equipped with more than one Xeon Phi We also assume a NETMASK=255.255.255.0 (--netbits=24 in the script below) so if a different NETMASK for the bridge is used (i.e. the provisioning interface has a NETMASK different than 255.255.255.0) then the script needs to be modified with either a hard-coded value or enhanced with a function that will do the MASK conversion from the value in the /etc/profile.nii to the value needed by micctrl)
DEBUG=1
LOGFILE=/tmp/bridge_log.txt
export BOOTIP=`grep NII_BOOTIP /etc/profile.nii | awk '{print $2}'|awk -F'=' '{print $2}'`
export MASK=`grep $BOOTIP /etc/profile.nii | grep NICDEF | awk '{print $2}'| awk -F'=' '{print $2}' | awk -F'|' '{print $3}'`
export DEV=`grep $BOOTIP /etc/profile.nii | grep eth | awk '{print $2}' | awk -F'|' '{print $1}'|awk -F'"' '{print $2}'`
export INST_IP=`grep NII_INSTALLERS /etc/profile.nii | awk -F'"' '{print $2}'`
date >> $LOGFILE 2>&1
ifdown $DEV
mv /etc/rc.kusu.d/S10NICStartup.rc* /root/
mv /etc/sysconfig/network-scripts/ifcfg-$DEV /root/
cat << EOF > /etc/sysconfig/network-scripts/ifcfg-$DEV
DEVICE=$DEV
ONBOOT=yes
NM_CONTROLLED=no
BRIDGE="br0"
MTU=9000
EOF
ifup $DEV
chmod 000 /opt/kusu/lib/plugins/cfmclient/00-updateNICs.sh
chmod 000 /opt/kusu/sbin/updatenic
cat << EOF > /etc/sysconfig/network-scripts/ifcfg-br0
DEVICE=br0
ONBOOT=yes
BOOTPROTO=static
IPADDR=$BOOTIP
NETMASK=$MASK
NM_CONTROLLED=no
TYPE=Bridge
EOF
if [ DEBUG ]
then
echo "ifconfig" >> /tmp/log.txt 2>&1
ifconfig >> /tmp/log.txt 2>&1
echo "addbr br0" >> /tmp/log.txt 2>&1
fi
#brctl addbr br0 >> /tmp/log.txt 2>&1
if [ DEBUG ]
then
echo "ifconfig" >> /tmp/log.txt 2>&1
ifconfig >> /tmp/log.txt 2>&1
echo "brctl addif br0 eth0" >> /tmp/log.txt 2>&1
fi
#brctl addif br0 eth0 >> /tmp/log.txt 2>&1
if [ DEBUG ]
then
echo "ifconfig" >> /tmp/log.txt 2>&1
ifconfig >> /tmp/log.txt 2>&1
fi
if [ DEBUG ]
then
echo "ifconfig br0 $BOOTIP netmask $MASK up" >> /tmp/log.txt 2>&1
fi
#ifconfig br0 $BOOTIP netmask $MASK up >> /tmp/log.txt 2>&1
if [ DEBUG ]
then
echo "ifconfig" >> /tmp/log.txt 2>&1
ifconfig >> /tmp/log.txt 2>&1
fi
ifup $DEV
ifup br0
# bridge is ok, now time to setup the mics
/usr/sbin/micctrl --cleanconfig
cp -f /etc/hosts /etc/hosts.orig
HOSTNAME_BASE=`grep NII_HOSTNAME /etc/profile.nii | awk -F'=' '{print $2}'`
IPMIC=`grep $HOSTNAME_BASE /etc/hosts | grep mic | awk '{print $1}'`
#/usr/sbin/micctrl --shutdown --wait mic0
/usr/sbin/micctrl --initdefaults
cp -f /etc/hosts.orig /etc/hosts
/usr/sbin/micctrl --addbridge=br0 --type=External --ip=$BOOTIP --netbits=24 mic0
/usr/sbin/micctrl --network=static --bridge=br0 --ip=$IPMIC mic0
/usr/sbin/micctrl --addnfs="/home" --dir="/home" --server="$INST_IP"
/usr/sbin/micctrl --addnfs="/depot/shared/" --dir="/shared" --server="$INST_IP"
/usr/sbin/micctrl --resetconfig
chkconfig --add mpss
/etc/init.d/mpss start
#micctrl --boot --wait mic0
