Hadoop test case scenarios

This section describes the test cases that were run on the local Hadoop cluster with Hadoop Storage Tiering configuration.

Please refer to the Hadoop test case scenarios section on how to test and leverage Hadoop Storage Tiering with IBM Storage Scale.

Note: If the remote IBM Storage Scale cluster shared path is configured in the ViewFs mount table of the Local Native HDFS, then you do not need to give the full schema path. One just requires specifying the directory path value.

For example, running a MapReduce WordCount job with input from the local native HDFS cluster to generate output to the remote HDFS Transparency cluster will just need to specify the /path instead of hdfs://<namenode_host>:<port>/path full schema path.

sudo -u hdfs yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-
mapreduce-examples.jar wordcount /tmp/redhat-release /gpfs/mapred/wordcount_hdfs

[root@c16f1n03 ~]# sudo -u hdfs hdfs dfs -ls -R 
hdfs://c16f1n10.gpfs.net:8020/gpfs/mapred/wordcount_hdfs
-rw-r--r--   3 hdfs root          0 2019-01-07 02:23 
hdfs://c16f1n10.gpfs.net:8020/gpfs/mapred/wordcount_hdfs/_SUCCESS
-rw-r--r--   3 hdfs root         68 2019-01-07 02:23 
hdfs://c16f1n10.gpfs.net:8020/gpfs/mapred/wordcount_hdfs/part-r-00000