Apache Mahout

Apache Mahout is an open source project to create scalable, machine learning algorithms. Mahout operates in addition to Hadoop, which allows you to apply the concept of machine learning via a selection of Mahout algorithms to distributed computing via Hadoop. Mahout's core algorithms include recommendation mining, clustering, classification, and frequent item-set mining.

Before you begin

Ensure that the MapReduce framework in IBM® Spectrum Symphony is set to use Hadoop. For the supported versions of Hadoop see Supported distributed files systems for MapReduce or YARN integration. For the supported versions of Mabout that the MapReduce framework in IBM Spectrum Symphony has been qualified with, see Supported third-party applications for MapReduce.

The MapReduce framework in IBM Spectrum Symphony is qualified with Mahout 0.9.0 on Apache HDFS 2.7.2. For information on installing Mahout, refer to Mahout documentation.

About this task

Follow these steps to use Mahout with the MapReduce framework in IBM Spectrum Symphony:

Procedure

  1. Download and decompress the Mahout package: 
    # tar zxvf mahout-distribution-0.9.tar.gz
  2. Source the IBM Spectrum Symphony profile cluster environment files: 
    # source profile.platform
  3. Define the HADOOP_HOME environment: 
    # export HADOOP_HOME=/root/hadoop
  4. Navigate to the Mahout home directory.
  5. Modify the mahout script file, located under mahout_install_dir/bin, to enable Mahout to run with the mrsh utility in the MapReduce framework rather than with the Hadoop job submission script.
    Change line 250 from:
    exec "$HADOOP_BINARY" jar $MAHOUT_JOB $CLASS "$@"
    to:
    exec "$PMR_BINDIR/mrsh" jar $MAHOUT_JOB $CLASS "$@"
  6. Place the Mahout package locally on all compute nodes (under the same local path) or save the Mahout .jar files to a location that all the compute hosts can access.
  7. Append the location where Mahout jars are stored to the USER_CLASSPATH variable in pmr-env.sh, located under $PMR_HOME/conf/.
  8. Download the reuters21578.tar.gz data set and place it in the $MAHOUT_HOME directory: 
    # curl http://kdd.ics.uci.edu/databases/reuters21578/reuters21578.tar.gz -o $MAHOUT_HOME/reuters21578.tar.gz
  9. Run the example script: 
    sh cluster-reuters.sh 1 $MAHOUT_HOME/reuters21578.tar.gz