Topic
12 replies Latest Post - ‏2012-05-16T23:27:04Z by tsm_dude
tsm_dude
tsm_dude
10 Posts
ACCEPTED ANSWER

Pinned topic agent incomplete status

‏2012-04-26T05:57:54Z |
Have VM managers defined and OK (7.2.2.2 client and server )
Have agents installed on each guest.
Everything OK and working.
Then...
All agents running on 3 particular VM hosts are incomplete.
All other agents running on other hosts within the Vcentre are fine.

These incomplete agents worked, then were vmotioned to other host and now dont work.
host table populated with TLM_VM_VMware-50 87 88 blah for these agents.

Vmware on non-working host was patched to 4.1.0 (348481) to 4.1.0(502767)

yes - readonly account is setup and works.
no - dont use enabler script as I have vmmanger connections.

Anyone aware of a bug with vm patches ?
Updated on 2012-05-16T23:27:04Z at 2012-05-16T23:27:04Z by tsm_dude
  • MaksKowalik
    MaksKowalik
    78 Posts
    ACCEPTED ANSWER

    Re: agent incomplete status

    ‏2012-04-26T09:32:40Z  in response to tsm_dude
    Hello,

    A couple of questions:
    1. Is the VM Manager connection configured in ILMT server a connection to Virtual Center, or there are multiple connections (to every particular ESX host)?
    2. In case of VCenter, could you confirm that when you login (using vSphere Client) to the VC using the same credentials as in ILMT server, the problematic host together with its (recently VMotioned) VMs is visible?

    Best regards,
    Maks Kowalik

    The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
    • tsm_dude
      tsm_dude
      10 Posts
      ACCEPTED ANSWER

      Re: agent incomplete status

      ‏2012-04-26T22:43:24Z  in response to MaksKowalik
      Hi Maks,

      1. Its a virtual centre connection. I did not define hosts directly as you then need to setup a read-only linux account. I use an account;
      DOMAIN\BL - IT Virtual Center Read-Only
      Read-only

      2. When I use the above account to login to VC I can see all the hosts, including the problematic hosts and working ones. In fact I see all hosts as expected.

      Its not a setup problem, as they worked previously. I suspect the ILMT DB has issues, and are attempting to find the tables effected. Or its the vmware patch.
      Do you know if VM 4.1.0(502767) has issues working with ILMT ?

      I am going to vmotion bad one back to a host that is visible and vmotion working one to bad host, see what happens. I suspect issue will follow the
      host.
      • MaksKowalik
        MaksKowalik
        78 Posts
        ACCEPTED ANSWER

        Re: agent incomplete status

        ‏2012-05-08T11:27:04Z  in response to tsm_dude
        Hi,

        I wouldn't suspect the VMware patch as the ILMT server talks only to the VCenter not to the problematic host (but everythings possible).
        If you want to investigate this by yourself please look at the content of tables in VIRT schema:
        ROOT (contains record of VC), CLUSTER, HOST, GUEST. The agent uploads the hardware scan with UUID of virtual machine it is running in. Then the server:
        • looks for this UUID among all the guests' UUIDs in the data retrieved from the VCenter
        • looks for the connection between guest, host, cluster
        If it cannot find the UUID or the UUID can be found but cannot be matched to one of the hosts, the agent stays in "Incomplete" state. The cluster-host-guest connection is verified by their UUIDs.

        Best regards,
        Maks Kowalik


        The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
        • tsm_dude
          tsm_dude
          10 Posts
          ACCEPTED ANSWER

          Re: agent incomplete status

          ‏2012-05-10T01:11:19Z  in response to MaksKowalik
          Hi Maks,

          thanks for the reply. Yes I checked those tables and there is no entries. So the root has both vmmanagers ok, but the cluster does not contain the rows, the host table is missing the hosts that contain the incomplete agents. The guests are also not there. The only thing that is there is the hostname of the incomplete guest in the agent table.
          I was going to try manually inserting the rows as I can see working hosts, and just copy those rows in, but the IDs I am unsure about. (ID and EXTERNAL_ID) I cant see any UUID fields as you mentioned.

          Do you any idea how the ID numbers are generated and how to fudge it ?

          thanks

          Steve
          • tsm_dude
            tsm_dude
            10 Posts
            ACCEPTED ANSWER

            Re: agent incomplete status

            ‏2012-05-10T02:56:49Z  in response to tsm_dude
            Here is the cluster table query output;

            15796859 FDC V4.1/DEV FDC V4.1/DEV VSphere 15796551
            15796957 FDC V4.1/DMZ FDC V4.1/DMZ VSphere 15796551
            15797402 GSDC/ORACLE GSDC/ORACLE VSphere 15797401
            15797433 GSDC/NON-PROD GSDC/NON-PROD VSphere 15797401
            15797508 GSDC/PROD GSDC/PROD VSphere 15797401
            15797677 GSDC/DMZ GSDC/DMZ VSphere 15797401
            15797703 GSDC/DEV GSDC/DEV VSphere 15797401
            15796552 FDC V4.1/ORACLE FDC V4.1/ORACLE VSphere 15796551
            15796590 FDC V4.1/NON-PROD FDC V4.1/NON-PROD VSphere 15796551
            15796691 FDC V4.1/PROD FDC V4.1/PROD VSphere 15796551

            And I run the query again and all IDs have changed, in a matter of seconds.
            15799227 GSDC/DMZ GSDC/DMZ VSphere 15798951
            15799253 GSDC/DEV GSDC/DEV VSphere 15798951
            15798102 FDC V4.1/ORACLE FDC V4.1/ORACLE VSphere 15798101
            15798140 FDC V4.1/NON-PROD FDC V4.1/NON-PROD VSphere 15798101
            15798241 FDC V4.1/PROD FDC V4.1/PROD VSphere 15798101
            15798409 FDC V4.1/DEV FDC V4.1/DEV VSphere 15798101
            15798507 FDC V4.1/DMZ FDC V4.1/DMZ VSphere 15798101
            15798952 GSDC/ORACLE GSDC/ORACLE VSphere 15798951
            15798983 GSDC/NON-PROD GSDC/NON-PROD VSphere 15798951
            15799058 GSDC/PROD GSDC/PROD VSphere 15798951

            Looks like agents have been all installed from 1 build and not cloned (tlmagent -clone)
            so the ids are all duplicated, causing all VIRT tables to be screwed.

            WARNING to all: Dont roll out agents from 1 build without cloning.
            • MaksKowalik
              MaksKowalik
              78 Posts
              ACCEPTED ANSWER

              Re: agent incomplete status

              ‏2012-05-10T12:06:14Z  in response to tsm_dude
              Hi,

              now I'm a bit confused - are the tables in VIRT empty or not?

              Manual insert into CLUSTER, HOST, GUEST will have no effect as the server cleans them every time before communication with VCenter and - just after the successful data extraction - merges them with data sent by agent. The IDs are generated every time during extraction; that's why you noticed them to change.
              The IDs are only for the virtualization topology building purpose.

              Data in VIRT schema appear ONLY as the result of ILMT server -> VCenter communication. Any problem with the agents (like the non-unique agent ids) will not damage them.

              The data merging between information from an agent and a VCenter works like that (simplified):
              • an agent uploads capacity scan containing the UUID of the virtual machine
              • server extracts the structure visible from VCenter and stores it in the VIRT schema
              • server looks for the UUID (previously sent by the agent) in the VIRT.GUEST.EXT_ID
              If it cannot find it, the agent stays as incomplete,
              otherwise the server fills the missing data (in terms of PVU calculation) in some other tables using information stored in VIRT.HOST

              Best regards,
              Maks Kowalik


              The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
              • tsm_dude
                tsm_dude
                10 Posts
                ACCEPTED ANSWER

                Re: agent incomplete status

                ‏2012-05-10T23:55:44Z  in response to MaksKowalik
                Hi,

                sorry to confuse. No the tables are not empty. We have many vm clusters that are working in ILMT fine and status is not incomplete. The issue is 1 cluster, that contains
                3 hosts that contain 70 odd guests. All the guests agents are incomplete.
                The only difference is the 3 hosts(problematic) are at a higher VMware 4.1 patch level.

                I have double checked the VIRT schema tables and found ;

                root - ok contains 2 VMmanager we defined in ILMT
                cluster - ok contains all clusters even the problematic one (FDC V4.1/DEV)
                host - ISSUE does not contain any of the 3 hosts in problematic cluster, all other hosts are there.
                these are svppdev1,svppdev2 and svppdev3.

                HOST;
                15863203 36303336-3032-5347-4831-313458354459 VSphere 15863202 1 svppora9.bizlink.nsw.gov.au
                15863223 36303336-3032-5347-4831-313458354534 VSphere 15863202 1 svppora8.bizlink.nsw.gov.au
                15863241 35393538-3235-5347-4830-323858323745 VSphere 15863240 1 svppnpd2.bizlink.nsw.gov.au
                15863271 35393538-3235-5347-4830-32385832374e VSphere 15863240 1 svppnpd1.bizlink.nsw.gov.au
                15863299 35393538-3235-5347-4830-323858323742 VSphere 15863240 1 svppnpd3.bizlink.nsw.gov.au
                15863343 36303336-3032-5347-4831-313458354458 VSphere 15863342 1 svppprd4.bizlink.nsw.gov.au
                15863388 36303336-3032-5347-4831-313458354457 VSphere 15863342 1 svppprd3.bizlink.nsw.gov.au
                15863433 36303336-3032-5347-4831-313458354532 VSphere 15863342 1 svppprd2.bizlink.nsw.gov.au
                15863482 36303336-3032-5347-4831-313458354530 VSphere 15863342 1 svppprd1.bizlink.nsw.gov.au
                15863609 34353934-3833-434e-3738-323030324b42 VSphere 15863608 1 svppdmz4.bizlink.nsw.gov.au
                15863610 34353934-3833-5347-4837-333839444146 VSphere 15863608 1 svppdmz5.bizlink.nsw.gov.au
                15864053 34353934-3835-434e-3738-323530325045 VSphere 15864052 1 svppora4.bizlink.nsw.gov.au
                15864069 34353934-3835-434e-3738-32353033504d VSphere 15864052 1 svppora3.bizlink.nsw.gov.au
                15864084 2172afdf-5503-194a-9a62-f6236018e9cb VSphere 15864083 1 svppnpd5.bizlink.nsw.gov.au
                15864101 b9ae2179-11e7-4247-9d7c-ff1b8fffde18 VSphere 15864083 1 svppnpd7.bizlink.nsw.gov.au
                15864129 00000000-0000-0000-0000-000000000000 VSphere 15864083 1 svppnpd4.bizlink.nsw.gov.au
                15864139 49ffb0a3-3d01-c84d-bba7-6e5835ffe992 VSphere 15864083 1 svppnpd6.bizlink.nsw.gov.au
                15864159 36303336-3032-5347-4831-313458354536 VSphere 15864158 1 svppprd6.bizlink.nsw.gov.au
                15864195 36303336-3032-5347-4831-313458354533 VSphere 15864158 1 svppprd5.bizlink.nsw.gov.au
                15864238 36303336-3032-5347-4831-313458354535 VSphere 15864158 1 svppprd8.bizlink.nsw.gov.au
                15864283 36303336-3032-5347-4831-313458354531 VSphere 15864158 1 svppprd7.bizlink.nsw.gov.au
                15864329 34353934-3833-434e-3738-313130335953 VSphere 15864328 1 svppdmz2.bizlink.nsw.gov.au
                15864341 34353934-3833-434e-3738-323830335542 VSphere 15864328 1 svppdmz1.bizlink.nsw.gov.au
                15864355 36303336-3032-5347-4831-313458354454 VSphere 15864354 1 svppdev4.bizlink.nsw.gov.au
                15864371 36303336-3032-5347-4831-313458354456 VSphere 15864354 1 svppdev6.bizlink.nsw.gov.au
                15864386 36303336-3032-5347-4831-313458354453 VSphere 15864354 1 svppdev5.bizlink.nsw.gov.au
                GUEST - not sure, no hostnames etc. Any ideas ? Thanks very much for your input.
                • MaksKowalik
                  MaksKowalik
                  78 Posts
                  ACCEPTED ANSWER

                  Re: agent incomplete status

                  ‏2012-05-14T10:11:43Z  in response to tsm_dude
                  Hi,

                  in your situation the ILMT server:
                  1. for some reason is unable to extract the information about the 3 problematic hosts
                  2. as a consequence of 1 it also ignores information about VMs running on them since there are no useful hardware information for PVU purposes. The VMs running on these hosts will not be inserted into VIRT.GUEST.

                  I would do two checks:
                  1. Using the read-only credentials specified in ILMT login to the VCenter (using vSpehere client), highlight the problematic host and check whether you can see the hardware information on "Summary" tab, and VMs on "Virtual Machines" tab
                  2. Test if you still have this problem while using in ILMT an account with greater rights to VMware infrastructure.

                  If you are still unable to identify the cause then open a PMR, because I don't think it would be possible to investigate it this further this way.

                  Best regards,
                  Maks Kowalik


                  The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
                  • tsm_dude
                    tsm_dude
                    10 Posts
                    ACCEPTED ANSWER

                    Re: agent incomplete status

                    ‏2012-05-16T06:37:50Z  in response to MaksKowalik
                    SOLUTION FOUND

                    • Problem caused by hosts with blank serial numbers in the BIOS (HP blades come blank, you need to add into BIOS and save)

                    ILMT coded to only handle 1 of these hosts as the blank ID is converted to all zeros in the host (VIRT schema)table.
                    So when another host with blank serial is encountered, ILMT inserts a duplicate row, then deletes it.
                    So your hosts never stay in DB and the guest end up incomplete.

                    This is the error;

                    <Trace Level="MIN">
                    <Time Millis="1337126256097"> 2012-05-16 09:57:36.097+10:00</Time>
                    <Server Format="IP">blah.company.au</Server>
                    <ProductId>COD</ProductId>
                    <Component>Admin Server</Component>
                    <ProductInstance></ProductInstance>
                    <LogText><![CDATA[LOG]: Deleted 3 hosts]></LogText>
                    <Source FileName="com.ibm.license.mgmt.bl.virt.PreprocessRulesBO" Method="deleteDuplicatedHosts"/>
                    <Thread>LMTTimerManager3.Alarm Pool : 0</Thread>

                    SOLUTION:
                    vmotion hosts off blank s/n BIOS machines, put s/n into BIOS and vmotion guests back, remove/install agent/ plugin
                    hw scan etc then all good.

                    Luckily we have an engineer working on some HP software used for hardware inventory, server info etc and he knew the serial
                    numbers were not there and could put 2 and 2 together

                    thanks all

                    Steve
  • tsm_dude
    tsm_dude
    10 Posts
    ACCEPTED ANSWER

    Re: agent incomplete status

    ‏2012-05-16T06:39:54Z  in response to tsm_dude
    ILMT does not like machines with blank serial number is BIOS.
    • MaksKowalik
      MaksKowalik
      78 Posts
      ACCEPTED ANSWER

      Re: agent incomplete status

      ‏2012-05-16T10:27:08Z  in response to tsm_dude
      Hi,

      ILMT is not interested in S/N in this case.
      ILMT server builds the VMware topology based on UUID numbers provided by VMware.
      It's VMware which is producing non unique numbers (as you noticed because they are generated from S/N).
      Another host which might have this problem is svppnpd4.bizlink.nsw.gov.au

      Best regards,
      Maks Kowalik


      The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
      • tsm_dude
        tsm_dude
        10 Posts
        ACCEPTED ANSWER

        Re: agent incomplete status

        ‏2012-05-16T23:27:04Z  in response to MaksKowalik
        Hi,

        I`m not trying to indicate which product has a fault, or causes this issue. The fix is all I am
        interested in. If you have more than 1 host with blank serial in BIOS they WILL NOT WORK in ILMT
        and guest agents will stay incomplete.

        svppnpd4 has blank serial, but its guests are COMPLETE since its the first host to be inserted, it does NOT have the problem.
        It is subsequent hosts with blank serials that will not work and guest on these will stay incomplete.

        query the host table and if you have all zeroes in the ID for a host, you may have this issue.
        FIX -> put serial into BIOS.

        Well, now working on new issue with reports that fail to sign. it never ends. ;)

        Thanks anyway.
        Steve