Topic
  • 9 replies
  • Latest Post - ‏2013-02-25T23:22:06Z by SystemAdmin
thanh
thanh
36 Posts

Pinned topic ISD 6.3 is not seeing Common Agent or Platform Agent on other servers

‏2013-02-08T16:17:11Z |
This is probably a tricky question: What should I look for in debugging the problem that ISD does not see Common Agent or Platform Agent on other servers? These servers have been discovered and taken full inventories by ISD and they're accessible.

But, when I go to the Group by Agents, these servers are listed in "Systems with No Agent". What could be wrong?

I manually installed Platform Agent and Common Agent on these servers and they are ACTIVE. I also imported the Platform Agent and Common Agent packages into ISD and did Install Agents with the wizard.

Where should I get started looking into?
Updated on 2013-02-25T23:22:06Z at 2013-02-25T23:22:06Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    13279 Posts

    Re: ISD 6.3 is not seeing Common Agent or Platform Agent on other servers

    ‏2013-02-13T15:57:52Z  
    Look for blocked ports/protocols. Also look for things like duplicate UUIDS/SSH keys, or cloned OS images.

    ----
    Craig Elliott
    IBM Advanced Technical Skills
  • bassemir
    bassemir
    39 Posts

    Re: ISD 6.3 is not seeing Common Agent or Platform Agent on other servers

    ‏2013-02-13T16:24:27Z  
    Look for blocked ports/protocols. Also look for things like duplicate UUIDS/SSH keys, or cloned OS images.

    ----
    Craig Elliott
    IBM Advanced Technical Skills
    I have seen this in my environment and resolved it by making sure I had the correct agent for the OS on the system as well as making sure a firewall was not blocking traffic.

    Here are a couple of suggestions.

    Look at the target system that was discovered and inventoried. Is it using just SSH protocol or it is using CIM or CAS protocols as well (CAS is used by common agent, CIM used by platform agent).

    If it is a common agent, look at /opt/ibm/director/agent/runtime/agent/config/endpoint.properties and grep for unmanaged (should be false) you should also see your ISD IP address listed several times in that file.

    If it is a platform agent, are you sure you are running the right agent for the right OS and I believe (not 100% sure) that the agent can not be a higher level than ISD. If you are running Redhat with KVM there is a different agent for that than just plain old Redhat (no KVM).

    Last resort would be to remove the target system OS and server MEPs from ISD and rediscover them. I have done this when I noticed a protocol missing (e.g. the CIM or CAS protocols).

    Rich
  • thanh
    thanh
    36 Posts

    Re: ISD 6.3 is not seeing Common Agent or Platform Agent on other servers

    ‏2013-02-21T18:49:29Z  
    • bassemir
    • ‏2013-02-13T16:24:27Z
    I have seen this in my environment and resolved it by making sure I had the correct agent for the OS on the system as well as making sure a firewall was not blocking traffic.

    Here are a couple of suggestions.

    Look at the target system that was discovered and inventoried. Is it using just SSH protocol or it is using CIM or CAS protocols as well (CAS is used by common agent, CIM used by platform agent).

    If it is a common agent, look at /opt/ibm/director/agent/runtime/agent/config/endpoint.properties and grep for unmanaged (should be false) you should also see your ISD IP address listed several times in that file.

    If it is a platform agent, are you sure you are running the right agent for the right OS and I believe (not 100% sure) that the agent can not be a higher level than ISD. If you are running Redhat with KVM there is a different agent for that than just plain old Redhat (no KVM).

    Last resort would be to remove the target system OS and server MEPs from ISD and rediscover them. I have done this when I noticed a protocol missing (e.g. the CIM or CAS protocols).

    Rich
    Thanks for all the helpful suggestions! This is the Platform Agent. The key factor for my case is probably the firewalls. Before the ISD discovery, I needed to stop both v4 and v6 firewalls. And, yes, it's a good practice to remove the OS object from ISD.

    If this is indeed the problem that ISD discovery is blocked by the firewalls, it would be helpful for debugging if an error or warning message is generated, as opposed to completing discovery with no error.

    -Thanh Lam.
  • SystemAdmin
    SystemAdmin
    13279 Posts

    Re: ISD 6.3 is not seeing Common Agent or Platform Agent on other servers

    ‏2013-02-22T16:59:27Z  
    • thanh
    • ‏2013-02-21T18:49:29Z
    Thanks for all the helpful suggestions! This is the Platform Agent. The key factor for my case is probably the firewalls. Before the ISD discovery, I needed to stop both v4 and v6 firewalls. And, yes, it's a good practice to remove the OS object from ISD.

    If this is indeed the problem that ISD discovery is blocked by the firewalls, it would be helpful for debugging if an error or warning message is generated, as opposed to completing discovery with no error.

    -Thanh Lam.
    We continue to search for ways to improve this. When discovering a system, it's impossible for Systems Director to know if discovery failed because there is a firewall blocking traffic, because the system is offline, or if there simply isn't a system at the specified IP address.

    ----
    Craig Elliott
    IBM Advanced Technical Skills
  • thanh
    thanh
    36 Posts

    Re: ISD 6.3 is not seeing Common Agent or Platform Agent on other servers

    ‏2013-02-22T18:57:12Z  
    We continue to search for ways to improve this. When discovering a system, it's impossible for Systems Director to know if discovery failed because there is a firewall blocking traffic, because the system is offline, or if there simply isn't a system at the specified IP address.

    ----
    Craig Elliott
    IBM Advanced Technical Skills
    Discovering a system in general is fine in what you described. But, in the case of discovering Platform Agent or Common Agent, the same discovering criteria is too vague. In this specific case, the user's aim is a particular agent. Is it possible to provide some discovery options like check box for Platform Agent or Common Agent? This way, ISD has a discovery purpose and may be able to return with a success or failure.

    Note, I wasn't sure what caused the ISD to see or not to see the Platform Agent in my previous post. I'm now seeing similar problem on a different system. Even with all the troubles through the first system, I still have no clue why the second system is in the same unknown conditions. And, I've tried all the tricks I've just learned from the first system. Are there any debug procedures...

    -Thanh Lam.
  • bassemir
    bassemir
    39 Posts

    Re: ISD 6.3 is not seeing Common Agent or Platform Agent on other servers

    ‏2013-02-22T20:08:40Z  
    • thanh
    • ‏2013-02-22T18:57:12Z
    Discovering a system in general is fine in what you described. But, in the case of discovering Platform Agent or Common Agent, the same discovering criteria is too vague. In this specific case, the user's aim is a particular agent. Is it possible to provide some discovery options like check box for Platform Agent or Common Agent? This way, ISD has a discovery purpose and may be able to return with a success or failure.

    Note, I wasn't sure what caused the ISD to see or not to see the Platform Agent in my previous post. I'm now seeing similar problem on a different system. Even with all the troubles through the first system, I still have no clue why the second system is in the same unknown conditions. And, I've tried all the tricks I've just learned from the first system. Are there any debug procedures...

    -Thanh Lam.
    If you go to your "Resource Explorer" find your second system (the one with no agent, in the eyes of System Director) and click on it, what Protocols do you see listed? CIM is probably not listed. Dor some reason ISD has not discovered the CIM protocol.

    Are you sure this problem system has the Platform agent running? Do you see any errors in its log?

    Is the problem server on the same IP network? You may have a network hardware doing some filtering.

    The bottom line is if the platform agent is not showing up in ISD and you know it is running, there is most likely a communication problem between them. I would do problem analysis on the network between the two of them.

    Rich
  • thanh
    thanh
    36 Posts

    Re: ISD 6.3 is not seeing Common Agent or Platform Agent on other servers

    ‏2013-02-22T21:19:51Z  
    • bassemir
    • ‏2013-02-22T20:08:40Z
    If you go to your "Resource Explorer" find your second system (the one with no agent, in the eyes of System Director) and click on it, what Protocols do you see listed? CIM is probably not listed. Dor some reason ISD has not discovered the CIM protocol.

    Are you sure this problem system has the Platform agent running? Do you see any errors in its log?

    Is the problem server on the same IP network? You may have a network hardware doing some filtering.

    The bottom line is if the platform agent is not showing up in ISD and you know it is running, there is most likely a communication problem between them. I would do problem analysis on the network between the two of them.

    Rich
    There's no CIM protocol listed except SSH. So we know this is the sign of ISD not seeing PA. But, based on what? Communications between the two systems seem to be fine.

    Out of desperation, I uninstalled PA and made sure that those rpms mentioned in the doc. were not there:

    yum -y erase tog-pegasus libcmpiutil libvirt-cim sblim-cmpi-nfsv3
    sblim-cmpi-fsvol sblim-gather-provider sblim-gather sblim-cmpi-base openslp

    After installation, made sure PA is running:

    root@NFS-server init.d# /etc/init.d/ibmpawatchdogd_x86_64 status
    ibmpawatchdog running

    And, ISD still doesn't see PA (after removing the OS MEP and rediscovering/inventorying). I did look for logs in /opt/ibm/platform/log/ this time. There were some error messages in ibm-ipmi.log:

    22 Feb 13 22:08:08 Info IPMI Instrumentation Library 1.3.0
    22 Feb 13 22:08:08 SysApi OpenIPMI: open(/dev/ipmi0): No such file or directory (2)
    22 Feb 13 22:08:08 SysApi OpenIPMI: No device found
    22 Feb 13 22:08:08 Error SP6 Load - libibmsp6.so: cannot open shared object file: No such file or directory
    22 Feb 13 22:08:08 SysApi No driver located
    22 Feb 13 22:08:08 Thread stop - Not started
    22 Feb 13 22:08:08 Info Library unloaded
    Does this look anything related? Verified that /dev/ipmi0 is not on this system. However, it's on the first system that PA has started to work. Maybe this is a clue. So my next question is: How is /dev/ipmi0 created in the first place?
  • thanh
    thanh
    36 Posts

    Re: ISD 6.3 is not seeing Common Agent or Platform Agent on other servers

    ‏2013-02-24T21:36:28Z  
    • thanh
    • ‏2013-02-22T21:19:51Z
    There's no CIM protocol listed except SSH. So we know this is the sign of ISD not seeing PA. But, based on what? Communications between the two systems seem to be fine.

    Out of desperation, I uninstalled PA and made sure that those rpms mentioned in the doc. were not there:

    yum -y erase tog-pegasus libcmpiutil libvirt-cim sblim-cmpi-nfsv3
    sblim-cmpi-fsvol sblim-gather-provider sblim-gather sblim-cmpi-base openslp

    After installation, made sure PA is running:

    root@NFS-server init.d# /etc/init.d/ibmpawatchdogd_x86_64 status
    ibmpawatchdog running

    And, ISD still doesn't see PA (after removing the OS MEP and rediscovering/inventorying). I did look for logs in /opt/ibm/platform/log/ this time. There were some error messages in ibm-ipmi.log:

    22 Feb 13 22:08:08 Info IPMI Instrumentation Library 1.3.0
    22 Feb 13 22:08:08 SysApi OpenIPMI: open(/dev/ipmi0): No such file or directory (2)
    22 Feb 13 22:08:08 SysApi OpenIPMI: No device found
    22 Feb 13 22:08:08 Error SP6 Load - libibmsp6.so: cannot open shared object file: No such file or directory
    22 Feb 13 22:08:08 SysApi No driver located
    22 Feb 13 22:08:08 Thread stop - Not started
    22 Feb 13 22:08:08 Info Library unloaded
    Does this look anything related? Verified that /dev/ipmi0 is not on this system. However, it's on the first system that PA has started to work. Maybe this is a clue. So my next question is: How is /dev/ipmi0 created in the first place?
    Just thought of something: Since IPMI is for monitoring and querying hardware status, can it work in a Virtual Machine? In other words, can Platform Agent for KVM work in a VM?

    Found these messages about IPMI in the PA install log:

    ================================================================================
    IBM Systems Director Platform Agent 6.3 installation.

    Extracting RPM files to /tmp/platform.4jEUdk9USB
    .....................................
    91266 blocks
    Preparing packages for installation...
    ################################################################################
    Starting the IBM Systems Director Platform Agent 6.3 Installation Transaction
    on: Thu Feb 21 22:53:19 CET 2013
    ################################################################################
    The system you are installing on does not have the IPMI (Intelligent Platform
    Management Interface) utilities installed. To install the IPMI utilities,
    install the OpenIPMI package using your distribution's package management
    system.

    Following that suggestion, I installed the OpenIPMI package. Any instructions on how to start the IPMI driver would be appreciated.

    -Thanh Lam.
  • SystemAdmin
    SystemAdmin
    13279 Posts

    Re: ISD 6.3 is not seeing Common Agent or Platform Agent on other servers

    ‏2013-02-25T23:22:06Z  
    • thanh
    • ‏2013-02-24T21:36:28Z
    Just thought of something: Since IPMI is for monitoring and querying hardware status, can it work in a Virtual Machine? In other words, can Platform Agent for KVM work in a VM?

    Found these messages about IPMI in the PA install log:

    ================================================================================
    IBM Systems Director Platform Agent 6.3 installation.

    Extracting RPM files to /tmp/platform.4jEUdk9USB
    .....................................
    91266 blocks
    Preparing packages for installation...
    ################################################################################
    Starting the IBM Systems Director Platform Agent 6.3 Installation Transaction
    on: Thu Feb 21 22:53:19 CET 2013
    ################################################################################
    The system you are installing on does not have the IPMI (Intelligent Platform
    Management Interface) utilities installed. To install the IPMI utilities,
    install the OpenIPMI package using your distribution's package management
    system.

    Following that suggestion, I installed the OpenIPMI package. Any instructions on how to start the IPMI driver would be appreciated.

    -Thanh Lam.
    The Systems Director Server uses SLP (port 427) or CIM (ports 5988, 5989 or 15988 or 15989) to discover the Platform Agents. If these ports are blocked, or if the SLP registration is not working, then the Server will not be able to discover the Agent. You can use slpquery to display what the Server can query via SLP.

    ----
    Craig Elliott
    IBM Advanced Technical Skills