Setting configuration options in CES HDFS

This section lists the steps to set the configuration options in the CES HDFS.

To set configurations in the CES HDFS environment, run the following steps:

Stop HDFS Transparency.
Get the configuration file that you want to change.
Update the configuration file.
Import the file to CES HDFS.
Upload the changes to CES HDFS.
Start HDFS Transparency.

Setting up the gpfs.ranger.enabled field

From HDFS Transparency 3.1.1-3, ensure that the gpfs.ranger.enabled field is set to scale. The scale option replaces the original true/false values.

Stop HDFS Transparency.
If you are using CDP Private Cloud Base, stop HDFS Transparency from the Cloudera Manager GUI. Otherwise, on the CES HDFS Transparency, run the following:
```
/usr/lpp/mmfs/bin/mmces service stop hdfs -a
/usr/lpp/mmfs/hadoop/sbin/mmhdfs hdfs-dn stop
```
After HDFS Transparency has completely stopped, on the CES HDFS node, run the following command:
```
/usr/lpp/mmfs/hadoop/sbin/mmhdfs hdfs status
```
Update the HDFS Transparency configuration files and upload the changes. Get the config files by running the following commands:
```
mkdir /tmp/hdfsconf
/usr/lpp/mmfs/hadoop/sbin/mmhdfs config export /tmp/hdfsconf   gpfs-site.xml
cd /tmp/hdfsconf/
```
Update the config files in /tmp/hdfsconf with the following changes:
```
<property> 
<name>gpfs.ranger.enabled</name> 
<value>scale</value> 
<final>false</final> 
</property>
```
Note: From HDFS Transparency 3.1.0-6 and 3.1.1-3, ensure that the gpfs.ranger.enabled field is set to scale. The scale option replaces the original true/false values.
Import the files into the CES HDFS cluster by running the following command:
```
/usr/lpp/mmfs/hadoop/sbin/mmhdfs config import /tmp/hdfsconf gpfs-site.xml
```
Upload the changes to the CES HDFS cluster by running the following command:
```
/usr/lpp/mmfs/hadoop/sbin/mmhdfs config upload
```
Start HDFS Transparency.
If you are using CDP Private Cloud Base, start HDFS Transparency from the Cloudera Manager GUI. Click IBM Spectrum Scale > Actions > Start.
Otherwise, on the CES HDFS Transparency node, run the following:
```
/usr/lpp/mmfs/bin/mmces service start hdfs -a
/usr/lpp/mmfs/hadoop/sbin/mmhdfs hdfs-dn start
```
After HDFS Transparency has completely started, on the CES HDFS node, run the following command:
```
/usr/lpp/mmfs/hadoop/sbin/mmhdfs hdfs status
```

Setting the Java heap size for NameNode/DataNode

HDFS Transparency does not set the Java™ heap size value in hadoop_env.sh for NameNode or DataNode. Therefore, the JVM autoscales based on the machine memory size.

If you need to set the Java heap size, perform the following:

Stop HDFS Transparency.
Ensure that HDFS Transparency has stopped by running the following command:
```
mmhdfs hdfs status
```

Get the config file by running the following command:

mkdir /tmp/hdfsconf /usr/lpp/mmfs/hadoop/sbin/mmhdfs config export /tmp/hdfsconf hadoop_env.sh cd /tmp/hdfsconf

In /tmp/hdfsconf, update the hadoop_env.sh to set the -Xmx and -Xms options for HDFS_NAMENODE_OPTS and/or HDFS_DATANODE_OPTS.

For example:

SHARED_HDFS_NAMENODE_OPTS=“-server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC 
-XX:ErrorFile=/var/log/hadoop/$USER/hs_err_pid%p.log -XX:NewSize=1248m -XX:MaxNewSize=1248m 
-Xloggc:/var/log/hadoop/$USER/gc.log-`date +‘%Y%m%d%H%M’` -verbose:gc -XX:+PrintGCDetails 
-XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:CMSInitiatingOccupancyFraction=70 
-XX:+UseCMSInitiatingOccupancyOnly -Xms9984m -Xmx9984m -Dhadoop.security.logger=INFO,DRFAS 
-Dhdfs.audit.logger=INFO,DRFAAUDIT”

export HDFS_NAMENODE_OPTS=“${SHARED_HDFS_NAMENODE_OPTS} 
-XX:OnOutOfMemoryError=\“/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node\” 
-Dorg.mortbay.jetty.Request.maxFormContentSize=-1 ${HDFS_NAMENODE_OPTS}”

export HDFS_DATANODE_OPTS=“-server -XX:ParallelGCThreads=4 -XX:+UseConcMarkSweepGC 
-XX:OnOutOfMemoryError=\“/usr/hdp/current/hadoop-hdfs-datanode/bin/kill-data-node\”
-XX:ErrorFile=/var/log/hadoop/$USER/hs_err_pid%p.log -XX:NewSize=200m -XX:MaxNewSize=200m 
-Xloggc:/var/log/hadoop/$USER/gc.log-`date +‘%Y%m%d%H%M’` -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xms1024m -Xmx1024m -Dhadoop.security.logger=INFO,DRFAS 
-Dhdfs.audit.logger=INFO,DRFAAUDIT ${HDFS_DATANODE_OPTS}
-XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly”

Note: You can set the -Xmx and -Xms options directly in the HDFS_NAMENODE_OPTS and HDFS_DATANODE_OPTS export options.

Import the files into the CES HDFS cluster.
Upload the changes to the CES HDFS cluster.
Start HDFS Transparency.
Check the status of the HDFS Transparency cluster by running the following command:
```
mmhdfs hdfs status
```