Troubleshooting the sensor

This topic describes common problems that occur with the VMware ESX computer system sensor and presents solutions for those problems.

Duplicate VMs are created

Problem

After discovery, certain VMs seem to have duplicates.

Solution

Agile Service Manager discovers two instances of a VM, one physical and one virtual. If they cannot be reconciled to the same specific machine, two instances can exist in the database with similar attributes. These are not duplicates, but two separately discovered instances of the same VM.

This distinction is key to troubleshooting the problem, and there are several things to check, starting with Agile Service Manager, moving into the VMware environment, and then finally troubleshooting the general network environment.

Issues related to a pre-existing instance or database

The first item to check when troubleshooting a reconciliation problem is the database. If the VM has made the transition to a new VM, the old VM might not be able to reconcile.

The old instance can be deleted, preferably before restarting the discovery. If multiple runs are necessary to try different solutions, remember to delete all instances of the existing VM in advance.

It can also help to delete the instance of the host ESX server. If it is feasible in the environment, it can be helpful to drop and re-create the Agile Service Manager database between discovery runs. Then run a new discovery and see if the duplicates still exist.

<primaryMACAddress> attribute

The main reason that two instances of a VM cannot be reconciled is that they have different values in the <primaryMACAddress> attributes. To determine this value for each instance, it is necessary to export the objects of type ComputerSystem from the Agile Service Manager database with the following command run on the Agile Service Manager server:

Non-Windows:

$COLLATION_HOME/sdk/bin/api.sh -u <username> -p <password> 
   find --depth 1 ComputerSystem > <filename>.xml

Windows:

%COLLATION_HOME%\sdk\bin\api -u <username> -p <password> 
   find --depth 1 ComputerSystem > <filename>.xml

An XML file that lists the first-level attributes for all instances of the ComputerSystem class is generated. Look for the short name of the duplicate instances and scroll down to the attribute named <primaryMACAddress>.

If the value is different for the two instances, it is necessary to troubleshoot the MAC address assignments in the configuration file on the server, on the VM itself, or both.

VM configuration

If a VM is configured in NAT or 'host only' mode, the VMwareESX sensor discovers the virtual instance, but the physical instance is not discovered.

VM configuration files on the ESX host server

The Agile Service Manager VMwareESX sensor gathers information from configuration files for each VM to be discovered. These configuration files can be located with the following ESX command:

vmware-cmd –l (this is a lower-case 'L')

This command lists the configuration file for each VM known to the ESX server, indicated by the .vmx extension.

These files are in XML format and are not case-sensitive. View the information in the configuration file for the VM that has duplicate instances.

Validate the information for each interface to ensure that the MAC address for each line corresponds to an interface on the VM itself.

ethernet0.present = "true"
ethernet0.networkName ="VM Network"
ethernet0.addressType = "generated"
ethernet0.generatedAddress="00:0c:29:c1:a5:ee"
ethernet0.generatedAddressOffset = "0"
ethernet1.present = "true"
ethernet1.networkName = "VM Network"
ethernet1.addressType = "generated"
ethernet1.generatedAddress="00:0c:29:c1:a5:f8"
ethernet1.generatedAddressOffset = "10"

If the values are different in the configuration file or on the VM, correct them and try the discovery again.

Configuration on the VM itself

On the VM, there is a command that displays the information for each networking interface.

On non-Windows systems, the command is ifconfig. On Windows the command is ipconfig.

Examine the output and validate the interface/MAC pairings against the ESX configuration file. You can also verify that each interface is working by pinging the associated IP address. Try the discovery again.

Recent changes in a VM or movement from one ESX server to another

If a VM has been migrated from one ESX server to another, it is possible that the configuration file was changed and that can affect discoveries.

If the lines that contain generatedAddress get deleted it can affect discoveries.

When migrating VMs in a VirtualCenter environment, any VM with a generated MAC address is going to change it. If there is an existing VM on the ESX server that can be successfully discovered, use the configuration file for that VM as an example and look for any lines that might have been deleted.

The ESX server where the VM originated can also be specified as a target in a scope to see if the VM discovers correctly on that ESX server. If there are lines that have been deleted or modified during migration, add or correct them and run the discovery again.

Name resolution

If the VM cannot resolve to a single machine on the network, it can end up in Agile Service Manager as two separate instances. If the VM has multiple interfaces, and all interfaces are visible on the network, multiple valid instances can be found. It might not be possible to merge all instances into a single instance.

This is typically caused by a mismatch between hosts files, DNS, NIS, or any other name resolution service.

The remedy is to test the name resolution by the machine short name a few times from the VM itself, the ESX server, and the Agile Service Manager server. All the responses must match. If they return different responses, modify the name service or hosts files until the results are consistent. Try the discovery again.

General network connectivity & routing

There are global networking factors to consider in troubleshooting Agile Service Manager discoveries. As it relates to VMware discoveries, a firewall or other networking consideration such as SSH might partially obscure discovery of either the ESX server or the VM.

If the VM is discovered correctly by the VMware sensor, it is displayed with only a short name label under the Physical Infrastructure: Overview > Systems Tier > Virtual Systems > VMware ESX

The VM itself appears only as an object under the heading of Other Ip Device or Other Computer System.

In the case where only the VM is discovered correctly by the OS sensor, it is displayed as the appropriate type of Computer System. The virtual instance is not displayed, and the ESX server might not be displayed either.

Correct the routing and firewall configuration until the Agile Service Manager server can ping and SSH to the ESX server and directly to each of the VMs, then try the discovery again.

Duplicate VMware ESX servers are created

Problem: VMware ESX servers (Version 2.5 (all releases)) seem to be duplicated. This problem occurs when a sequential discovery is run by using the VMware ESX computer system sensor followed by the VMware Virtual Center server sensor.
Solution: You must manually merge duplicated VMware ESX servers.