Monitoring MapReduce jobs

Once MapReduce jobs are submitted, monitor them from the cluster management console or from the command line.

Within the MapReduce framework in IBM® Spectrum Symphony, you can monitor jobs using the job ID or name (as specified at job creation). The job name is generally available on the cluster management console, with commands that return job-related information, and in history files. The name of a MapReduce job can include ASCII characters 32 to 126 but cannot include quotation marks ("), backslashes (\), or non-printing characters except spaces ( ). If the job ID does include quotation marks or backslashes, the system converts them to underscore characters (_). If the job ID includes non-printing characters, the system converts them to spaces ( ).

If you do not set a job name when you create your job, the name of the application JAR file is automatically applied. If you do not set a job name for Pipe jobs, no job name is applied.

Job monitoring with the cluster management console

You can monitor MapReduce jobs on the MapReduce console:
  1. From the Dashboard's Common Tasks menu, select MapReduce Workload.

    The console opens in a new window.

  2. Click MapReduce > Workload > Jobs.

    At this level, you can view details of a job such as job status and map/reduce task summary.

  3. Select a specific job to view detailed information.
  4. From the Tasks tab, click a specific task to view detailed task information, kill running tasks, or retrieve logs.

Job monitoring with the CLI

You can monitor MapReduce jobs through the CLI using commands such as soamview.

The information that soamview displays depends on your user ID authorization:
  • The cluster administrator can view all information.
  • The consumer administrator can view information for all applications that are associated with that consumer. To specify an absolute consumer name that contains spaces, enclose the name in double quotation marks ().
  • Consumer users can view information for only those applications that their user privileges allow.
Use this syntax to view information about jobs:
session application_name [-c create_time_interval] [-e end_time_interval] [-d] [-s "all|open|closed|suspended|aborted"] [-t session_tag] [-U submission_user] [-S job_name] [-n counter] [-u user_name] [-x password]

When searching for jobs, if your search contains the -S flag, the asterisk (*) and question mark (?) characters in the job name behave as wildcard characters. The asterisk matches zero or more characters, while the question mark matches a single character.

For example, to return all jobs with names that starts with job and end with name with multiple characters in between (such as jobname, job123name, and job*name), run:
soamview session MapReduce732 -s all -S "job*name"
To returns all jobs with names that start with job and end with name with a single character in between (such as job name and job?name), run:
soamview session MapReduce732 -s all -S "job?name"