IBM Monitoring Agent for Hadoop, 7.1.0-TIV-ITM_Hadoop-FP0000

Download Description

Copyright International Business Machines Corporation 2018.
All rights reserved.

Component: IBM(R) Monitoring Agent for Hadoop,
Version 7.10.0.0

Component ID: 5724C04HP

Interim Fix:- None

Date: May 15, 2018

Contents:

1.0 General description
2.0 Problems fixed
3.0 Architecture and prerequisites
4.0 Image directory contents
5.0 Installation instructions
6.0 Additional installation information
7.0 Known problems and workarounds
8.0 Additional product information
9.0 Notices


1.0 General description
=======================
The Monitoring Agent for Hadoop (product code H8) provides you with the capability to monitor Hadoop cluster.


2.0 Problems fixed
==================
The following problems are addressed by this Agent.

2.1 APARs
---------
None.

2.2 Defects
-----------
None.

2.3 Enhancements
----------------
None.

2.4 Superseded fixes
--------------------
None.

2.5 APARs and defects included from superseded fixes
-----------------------------------------------------
None.


3.0 Architecture and prerequisites
==================================
This Agent is supported on all operating systems listed in the IBM Monitoring Agent for Hadoop User's Guide, version 7.10.00.00.

Please refer the Tivoli operating system and application support matrix for the latest certification information at the section "IBM Support-> Tivoli Supported Platforms" of IBM Knowledge Center Documentation.


3.1 Prerequisites for this Agent:
---------------------------------
- Before installing and configuring the agent, make sure your environment meets the requirements for the Monitoring Agent for
Hadoop.
- For the most up-to-date information about system requirements, see the Software product compatibility reports on IBM official
website.

4.0 Image directory contents
============================
This Agent image contains the following files:
- 7.10.00.00-ITM-Hadoop-README - This README file
- H8V710_DVD_201804110536.tar.gz - Agent archive .tgz format

The Agent archive file contains the following files:
- install.sh
- AUTORUN.INF
- DeployLnk.sh
- hadoop_silent_config.txt
- kcirunas.cfg
- notices
- README.TXT
- silent_config.txt
- silent_install.txt
- Deploy
- InstallITM
- license
- REPORTS
- unix
- WINDOWS

5.0 Installation instructions
=============================

Use the following steps to install this agent.

5.1 Before installing the Agent
-----------------------------
- The prerequisites listed under section 3.1 entitled 'Prerequisites for this Agent' must be installed before this agent can be
installed.

- For the purpose of this README, the symbol <CANDLEHOME> is the IBM Tivoli Monitoring installation directory. The default value
for CANDLEHOME is '/opt/IBM/ITM' on UNIX systems and 'C:\IBM\ITM' on Windows systems.

- For detailed, up-to-date information about installation prerequisites, see the Software product compatibility reports (
http://publib.boulder.ibm.com/infocenter/prodguid/v1r0/clarity/index.html).

- For links to information about installation prerequisites, see the Prerequisites topic for the agent in the IBM Tivoli Composite
Application Manager for Applications Information Center (http://publib.boulder.ibm.com/infocenter/tivihelp/...).

- Refer to section 7.0 entitled "Known problems and workarounds" before installation.

5.2 Installation instructions for Hadoop Agent
-----------------------------------------------
The Agent Installer folder in this package contains the following installation scripts for Windows and Linux/AIX. These installation scripts are for installing Hadoop Agent as well as required TEMS and TEPS support for the agent.

setup.exe (for Windows)

install.sh (for Linux/AIX)


Steps for installing Hadoop Agent:
-----------------------------------
A. Installing agent on Windows Systems:

- Download the complete package to any drive/folder and extract the package.

- Open the setup.exe file present in the installer under Windows folder and run as administrator.

- Click next button on “Welcome to the installer for IBM Monitoring Agent for Hadoop” window.

- Click next button on “IBM Monitoring Agent for Hadoop Prerequisites” window.

- Accept the license agreement terms on “Software License Agreement” window.

- On the “Select Features” window expand the Tivoli Enterprise Monitoring Agents – TEMA and select Monitoring Agent for Hadoop
and click next.

- Click next button on “Start copying Files” window and select yes when prompted by the installsheild.

- Click next button on “Setup Type” window.

- Click “Cancel” on “Agent Advance Configuration” window.

- After installation installing process, follow the steps in Section 5.4 of this README to configure the agent.

Remember:- Above steps are for installation of Agent on local machine. Optionally, we can install the Agent remotely.



B. Installing agent on Linux/AIX Systems:

- Download the complete package to the Linux server and extract the package.

- Open the Putty client, go to the installer folder containing install.sh file and run.

- Enter the name of the IBM Tivoli Monitoring installation directory (default: /opt/IBM/ITM)

- Confirm the directory by pressing 1 for next prompt

- Enter the name of the IBM Tivoli Monitoring local data directory (default: /opt/IBM/ITM)

- Confirm the directory by pressing 1 for next prompt.

- Select the choice for product installation.

Remember:- Here selection of choice will be 1 i.e. Install products to the local host.

- Next select the choice for license agreement. Press 1 to accept the license.

- Press "ENTER" to accept the default Encryption Key

- Select choice as 1 for “Product packages available for underlined operating system and component support categories”.

- Confirm the above selection.

- Select the choice for product which you want to install.

Remember:- We should type 1 to install "Monitoring Agent for Hadoop V07.10.00.00 " . However, for installing framework, you should also install "Tivoli Enterprise Services User Interface Extensions V06.30.06.00"

- Confirm the selection choice.

- Select choice for prerequisite check. If the prerequisite check fails/succeeds, please proceed with the installation.

- Select choice for installation of additional products or product support packages.

- Select choice for securing the IBM Tivoli Monitoring installation.

- After the installing process completed, follow the steps in Section 5.5 of this README to configure the agent.

Remember:- Above steps are for installation of Agent on local machine. Optionally, we can install the Agent remotely.


5.3 Instructions for adding support of Hadoop Agent on TEMS and TEPS
---------------------------------------------------------------------
Prerequisite:- ITM components should be installed and should be in started state.

Steps for adding support of Hadoop Agent on TEMS and TEPS will have all the similar steps as mentioned in 'Section 5.2: Installation instructions for Hadoop Agent' with slight difference in particular step.

Windows:- Follow the steps mentioned in "Installing agent on Windows Systems:" of section 5.2 A and do the following selection on the “Select Features” window:-

- Expand the Tivoli Enterprise Monitoring Server – TEMS and select Monitoring Agent for Hadoop to add support on TEMS.
- Expand the Tivoli Enterprise Portal Server – TEPS and select Monitoring Agent for Hadoop to add support on TEPS
- Expand TEP Desktop Client-TEPD and select Monitoring Agent for Hadoop to add support on TEPD and click next.

Remember:- If you have distributed environment of ITM, in that case select the particular Feature to add Hadoop support on that component of ITM.


Linux/AIX:- Follow the steps mentioned in "Installing agent on Linux/AIX Systems:" of section 5.2 B and do the appropriate selection as per the ITM components available for “Product packages available for underlined operating system and component support categories”.

-Select choice as 2 to add Tivoli Enterprise Portal Browser Client support
-Select choice as 3 to add Tivoli Enterprise Portal Server support
-Select choice as 4 to add Tivoli Enterprise Monitoring Server support

After proper selection of your choice, follow the rest of steps.

5.4 Instructions for configuring Hadoop Agent on Windows
---------------------------------------------------------
You can configure the agent on Windows operating systems by using the Manage Tivoli Enterprise Monitoring Services windows.

- Click Start > All Programs > IBM Tivoli Monitoring > Manage Tivoli Monitoring Services

- Open Manage Tivoli Enterprise Monitoring Services window and right-click Monitoring Agent for Hadoop and go to Advanced.

- Click Configure Advanced.

- Attention: If Configure Advanced is disabled, click Reconfigure.

- Enter the IP Address/Host name of TEMS and click OK.

- Remember: Here, protocol for TEMS connection and corresponding port number for TEMS communication can be changed depending
upon how TEMS is configured.

By Default protocol type is IP: PIPE and corresponding Port number is 1918.

- On the Details of Hadoop Cluster tab, Enter hostname and appropriate port number for NameNode, ResourceManager,
JobHistoryServer and Additional NameNode

Remember: It is necessary to validate that hostnames which are being configured, can be resolved on the agent machine. Also, Additional Namenode can either be Secondary Namenode or Standby Namenode. If additional NameNode is Secondary then generally the default value of this port number for SecondaryNamenode daemon process is 50090. Otherwise, if additional NameNode is Standby then generally the default value of this port number for Standby Namenode daemon process is 50070.

- Select Yes if Standby ResourceManager is part of Hadoop cluster, otherwise select No.

Remember: Here, If yes is selected, Details of Standby ResourceManager must be provided in the Details of Standby ResourceManager tab.

- Select Yes for Monitoring of Hadoop Services for Ambari based Hadoop Installations, otherwise select No.

Remember: Here, If yes is selected, Details of Ambari Server must be provided in the Details of Ambari server tab.

- Click Next.

- On the Details of Ambari Server tab, Enter hostname and appropriate port number for Ambari Server, as well as user name and
password for Ambari user.

- Click Next.

- On the Java parameters tab, select a value for the Java trace level parameter from the Java list

- Keep JVM arguments field as blank. You can optionally provide any JVM options.

- Click Next.

- On the Details of Standby ResourceManager tab, Click new to Add standby ResourceManager.

- Enter Hostname and appropriate port number for Standby ResourceManager.

- Click Next.

- On Java API Client Configuration tab, Keep Class Path for external jars as blank. You can optionally provide any other jars.

- Click OK.

- Right-click Monitoring Agent for Hadoop and click Start.

Remember:- If the configuration for some of the hosts is changed, agent must be configured again.

5.5 Instructions for configuring Hadoop Agent on Linux/AIX
-----------------------------------------------------------

You can configure the agent on Linux/AIX operating systems using two ways i.e. by either opening Manage Tivoli Enterprise Monitoring Services window in GUI mode or using Command line.

For GUI Mode:

Open Manage Tivoli Enterprise Monitoring Services window and follow the above steps mentioned in Section 5.4.

Remember:- There is a minor difference in the sequence of windows that appear when configuring the agent on Linux/AIX environment. In Linux/AIX environment, the Hadoop hosts configuration page will appear before the TEMS connection configuration page.

For Command Line:-

- On the command line run the following command: install_dir/bin/itmcmd config –A h8 where install_dir is the installation
directory of Hadoop agent (same as CANDLEHOME path). The agent is installed at the following default installation directory: /opt/IBM/ITM

- When the command line displays the message: Edit Monitoring Agent for Hadoop settings, press enter or 1(Yes)

- User will now be asked to enter the Details of the Hadoop Cluster.

- When the command line displays the message: Hostname of the machine where Hadoop daemon process for NameNode
is running. NameNode Hostname (default is :), add the NameNode hostname and press enter.

- When the command line displays the following message: Port number associated with the NameNode daemon process.
Generally the default value of this port number for NameNode daemon process is 50070.
NameNode Port (default is: 50070), add the NameNode port and press enter.

- When the command line displays the message: Hostname of the machine where Hadoop daemon process for ResourceManager is running.
ResourceManager Hostname (default is :), add the ResourceManager hostname and press enter.

- When the command line displays the following message: Port number associated with the ResourceManager daemon process.
Generally the default value of this port number for ResourceManager daemon process is 8088.ResourceManager Port (default is :), add the ResourceManager port and press enter.

- When the command line displays the message: Hostname of the machine where Hadoop daemon process for JobHistoryServer is running.
JobHistoryServer Hostname (default is:),add the JobHistoryServer hostname and press enter.

- When the command line displays the message: Port number associated with the JobHistoryServer daemon process. Generally the
default value of this port number for JobHistoryServer daemon process is 19888.JobHistoryServer Port (default is:),add the JobHistoryServer port and press enter.

- When the command line displays the message: Hostname of the machine where Hadoop daemon process for Additional NameNode
(Secondary/Standby) is running. Additional NameNode Hostname (default is:),add the Additional NameNode hostname and press enter.

- When the command line displays the message: Port number associated with the Additional NameNode daemon process. If additional
NameNode is Secondary then generally the default value of this port number for SecondaryNamenode Daemon process is 50090.Otherwise, if additional NameNode is Standby then generally the default value of this port number for Standby Namenode daemon process is 50070. Additional NameNode Port (default is: ), add the Additional NameNode hostname and press enter.

- When the command line displays the following message, type 1 and press Enter to add Details of Standby ResourceManager.
Standby ResourceManager(s) in Hadoop Cluster [1=Yes, 2=No] (default is: 2):

Remember: If "1" is entered for monitoring the Standby ResourceManager(s) in Hadoop Cluster then only then the messages mentioned for entering Standby ResourceManager's hostname and port will be displayed on command line.

- When the command line displays the following message, type 1 and press Enter to add Details of Ambari Server.
Monitoring of Hadoop services for Ambari based Hadoop installations [1=Yes, 2=No] (default is: 2):

Remember: If "1" is entered for Monitoring of Hadoop services for Ambari based Hadoop installations only then the messages mentioned for entering Ambari hostname, port, username and password will be displayed on command line.

- When the command line displays the message: Hostname of the machine where daemon process for Ambari server is running.
Ambari server Hostname (default is: ), add the Ambari server hostname and press enter.

- When the command line displays the message: Port number associated with the Ambari server. Generally the default value of this
port number for Ambari server daemon process is 8080.Ambari server Port (default is: 8080), add the Ambari server port and press enter.

- When the command line displays the message: Username of Ambari user. Username of Ambari user (default is :), add the
Ambari server username and press enter.

- When the command line displays the message: Password of Ambari user. Enter Password of Ambari user (default is :), add the
Ambari server password and press enter.

- When the command line displays the following message, select the appropriate trace level and press Enter.
This parameter allows you to specify the trace level used by the Java providers Java trace level
[1=Off, 2=Error, 3=Warning, 4=Information, 5=Minimum Debug, 6=Medium Debug, 7=Maximum Debug, 8=All](default is: 2)

- When the command line displays the following message, keep it blank for the Java™ virtual machine and press Enter.
This parameter allows you to specify an optional list of arguments to the java virtual machine JVM arguments (default is:)

- When the command line displays the following message, type 1 and press Enter to add a host.
Edit 'Hadoop High Availability (HA) Cluster with Standby ResourceManagers' settings, [1=Add, 2=Edit, 3=Del, 4=Next,5=Exit] (default is: 5):

Remember:

- To add more hosts, type 1.

- To modify the configuration settings of a Standby ResourceManager host, type 4 and press Enter until you see the host
name. Once you reach the host, type 2 to edit.

- When you are prompted, enter the Host name of Standby ResourceManager that is being monitored and press Enter.

- When you are prompted, enter the Host name of the Standby ResourceManager and press Enter.

- Type 5 and press Enter to exit the configuration steps.

- When the command line displays the following message, Enter a class path that lists additional jars. Optionally you can
keep it blank and press Enter.

- Do the next steps to connect Agent to TEMS in similar way as it is done for Linux/AIX OS agent connection to TEMS.
Here, you can select default value or required value in each step.

- Once Agent connection to TEMS is completed, Run the following command to start the agent:
install_dir/bin/itmcmd agent start h8


5.6 Local agent update
-----------------------
- If you have a previous version of agent installed, then run the new installer by referring the steps mentioned in ‘Section
5.2: Installation instructions for Hadoop Agent’ to upgrade the Agent. Also, upgrade the support for TEMS and TEPS by referring the steps mentioned in ‘Section 5.3: Instructions for adding support of Hadoop Agent on TEMS and TEPS’.

- When you upgrade from the socket - based agent (6.3.0 Fix Pack 2, or earlier) to the REST API - based agent (6.4.0, or later),
you need to reconfigure the agent. For reconfiguring the agent, complete the configuration steps that are specified in ‘Section 5.4: Instructions for configuring Hadoop Agent on Windows’ and ‘Section 5.5: Instructions for configuring Hadoop Agent on Linux’. `It is necessary to provide configuration for NameNode, ResourceManager, JobHistory Server, Additional NameNode (Secondary/Standby) as well as Standby ResourceManager and Ambari Server. However, ensure that you specify the host names according to the following guidelines when you configure the agent.

a. The host name of various daemon processes (NameNode, ResourceManager, and so on) that you specify must be the same (case and format) as the host names that are configured for the socket - based agent.

b. The fully qualified domain name (FQDN) must be used when you specify a host name. For example, hos1.ibm.com. If the length of the FQDN exceeds 25 characters, specify only the short host name without the domain name. For example, if the FQDN of a host ismyhadoopclustersetupnode.ibm.com, the short host name is myhadoopclustersetupnode.

- After you configure the agent that is upgraded, revert the changes that were made in the Hadoop Cluster for the Hadoop agent by using following steps:-

a. To prevent generation of unnecessary logs, remove the 17-line code from the hadoop-metrics2.properties files of all the Hadoop nodes.(Refer the step 2(a) of procedure section given in ‘Configuring Hadoop nodes' section of the web documentation available in IBM Performance Management 8.1.3 -> Configuring your environment -> Configuring the Hadoop agent)

b. Stop the Hadoop services.

c. Delete the Plugin.jar file that was copied from the agent installer from all nodes in the Hadoop cluster.

d. Start the Hadoop services.


5.7 Remote agent update
------------------------
1. Transfer the appropriate archive file (H8V710_DVD_201804110536.tar.gz) to a temporary directory on the IBM Tivoli
Enterprise Monitoring Server system. For the purpose of this README, the symbol <TEMP> represents the fully qualified path to this directory.

2. Expand the archive file using the "tar" command on UNIX systems or an extract utility on Windows systems. This step creates a
directory structure that contains fixes for all of the supported platforms.

3. To add the agent fix bundles into the remote deploy depot, use the "tacmd addBundles" command found in $CANDLEHOME/bin on UNIX
systems or in %CANDLE_HOME%\bin on Windows systems. For more information on the "tacmd addBundles" command, see the IBM Tivoli
Monitoring Administrator's Guide.

On UNIX systems,
If the agent installer was extracted into <TEMP>/H8V710_DVD_201804110536:
> $CANDLEHOME/bin/tacmd addBundles -n -i <TEMP>/H8V710_DVD_201804110536

On Windows systems,
if the fix was expanded to <TEMP>\H8V710_DVD_201804110536:
> %CANDLE_HOME%\bin\tacmd addBundles -n -i <TEMP>\H8V710_DVD_201804110536

where:
-n indicates that prerequisite bundles are not automatically
added. The -n parameter must be used because the fix
directory does not contain any prerequisites that the fix
might require. Please see Section 3.1 for the prerequisites
for this fix.
-i is the directory that contains the deployment bundles to be
added to the depot.

4. To log in to the Tivoli Enterprise Monitoring server, and deploy the fix to the appropriate nodes where the agent is
running,use the following "tacmd" commands. For more information on the "tacmd login" and "tacmd updateAgent" commands, see the IBM Tivoli Monitoring Administrator's Guide.

On UNIX systems:
> $CANDLEHOME/bin/tacmd login -s <server>
-u <itmuser>
-p <password>

> $CANDLEHOME/bin/tacmd listSystems

The output shows the Managed System Name for the OS agent on the remote system to be updated. Use this value as the target of the
"tacmd updateAgent" command.

> $CANDLEHOME/bin/tacmd updateAgent -t H8
-n <Managed system name>
-v 07100000

On Windows systems:
> %CANDLE_HOME%\bin\tacmd login -s <server>
-u <itmuser>
-p <password>

> %CANDLE_HOME%\bin\tacmd listSystems

The output shows the Managed System Name for the OS agent on the remote system to be updated. Use this value as the target of the
"tacmd updateAgent" command.

> %CANDLE_HOME%\bin\tacmd updateAgent -t H8
-n <Managed system name>
-v 07100000

Note:
- The component (-t) for the "tacmd updateAgent" command is
specified as two characters (H8), not three characters (KH8).
- The node (-n) for the "tacmd updateAgent" command is the managed
system name of the operating system (OS) agent to be updated.
The target node for the "tacmd updateAgent" command is always
an OS agent.

5.8 Agent support update
-------------------------
We can upgrade support locally in the similar manner as we install support for the first time as given in Section 5.3

Silent installation and configuration option
--------------------------

On Linux/Unix:-
-------------------
1. Locate the silent_install.txt and silent_config.txt at the location where Hadoop Agent installer
"H8V710_DVD_201804110536.tar.gz" is extracted.
2. Follow the steps as mentioned in the silent_install.txt file for Agent installation.
3. Follow the steps as mentioned in the silent_config.txt file for Agent configuration.

On Windows:-
--------------
1. Locate the silent.txt in the Hadoop Agent installer for Windows platform in the "WINDOWS" folder, of the location where
Hadoop Agent installer "H8V710_DVD_201804110536.tar.gz" is extracted.
2. Follow the steps as mentioned in the silent.txt file for Agent installation and configuration.

6.0 Additional installation information
=======================================

6.1 Additional information on using "itmpatch" command
------------------------------------------------------
None

7.0 Known problems and workarounds
==================================

Please refer the Chapter 3 of the Monitoring Agent for Hadoop Troubleshooting Guide available on the IBM Knowledgecenter.
(https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=0587adbc-8477-431f-8c68-9226adea11ed#fullpageWidgetId=W42ce7c6afdb9_42c2_a9ea_e1ba310bea8c&file=4f3cf3b6-95a0-4bef-93de-83595695a855)


8.0 Additional product information
==================================
None


9.0 Notices
===========
This information was developed for products and services offered in the United States. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify
the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to:

IBM Director of Licensing
IBM Corporation
North Castle Drive
Armonk, NY 10504-1785
U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law:

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,INCLUDING,BUT NOT LIMITED TO,THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

Trademarks and service marks
-----------------------------

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many
jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.