Topic
  • 11 replies
  • Latest Post - ‏2012-06-20T11:34:04Z by SystemAdmin
SystemAdmin
SystemAdmin
92 Posts

Pinned topic Storage / compute nodes don't show up in admin console.

‏2012-06-01T11:59:15Z |
Hey all,

I've recently started with the installation of SmartCloud Provisioning.

Installed 2 physical storage nodes and one virtual compute node on ESXI 5.0 host.
After that, I installed the kernel-services en they are all running on the storage nodes.

They seem to be installed correctly... (but probably not)
I can logon into admin console / webconsole however, in the admin console, the nodes don't show up in the "Cloud infrastructure" tab.

Tried reinstalling kernel-services, rebooted compute nodes, ...

Did anyone had these kind of problems while installing SCP and does anyone has some proposals for how I can verify this?

Thanks! Jan
Updated on 2012-06-20T11:34:04Z at 2012-06-20T11:34:04Z by SystemAdmin
  • rossdavibm
    rossdavibm
    25 Posts

    Re: Storage / compute nodes don't show up in admin console.

    ‏2012-06-01T14:21:51Z  
    Questions: Are you running on AMD or Intel processors? From my limited work with ESX 5 (mostly via proxy helping a co-worker as he fumbled through) and some research, it seems AMD is handling nested virtualization a bit cleaner than Intel at this time for ESX. Again, just anecdotal evidence, not a defining statement by any means.

    Using physical hardware for the storage nodes should allow the HSLT infrastructure to build out as needed, especially since you are seeing all the kernel services VMs running.

    Have you tried any CLI commands? Connect to storage-1, run:

    source /iaas/cli/iaasrc
    cd /iaas/cli
    ./iaas-describe-nodes

    You should see the listing of the compute/storage nodes. If not, you should get more feedback to determine the issue. Often times, for me, restarting HBASE/DFS on hbase-1 (if running 3 Hbase servers), and restarting webservices clears any transient issues I have encountered.

    One thing to check is in the VMX file for your compute node. Are you using KVM as the hypervisor running in a VM on ESX, or are you using the ESX server itself as a compute node? If the former, VMware supports 'hot add' of CPUs, but it's been documented to cause issues with libvirtd, which might be contributing to your problem. Add/modify vcpu.hotadd = "FALSE" in the VMX for your compute node if this is the case.

    If you can check these things out and report back, we might be able to move you along. Good luck!
  • SystemAdmin
    SystemAdmin
    92 Posts

    Re: Storage / compute nodes don't show up in admin console.

    ‏2012-06-05T11:21:32Z  
    Questions: Are you running on AMD or Intel processors? From my limited work with ESX 5 (mostly via proxy helping a co-worker as he fumbled through) and some research, it seems AMD is handling nested virtualization a bit cleaner than Intel at this time for ESX. Again, just anecdotal evidence, not a defining statement by any means.

    Using physical hardware for the storage nodes should allow the HSLT infrastructure to build out as needed, especially since you are seeing all the kernel services VMs running.

    Have you tried any CLI commands? Connect to storage-1, run:

    source /iaas/cli/iaasrc
    cd /iaas/cli
    ./iaas-describe-nodes

    You should see the listing of the compute/storage nodes. If not, you should get more feedback to determine the issue. Often times, for me, restarting HBASE/DFS on hbase-1 (if running 3 Hbase servers), and restarting webservices clears any transient issues I have encountered.

    One thing to check is in the VMX file for your compute node. Are you using KVM as the hypervisor running in a VM on ESX, or are you using the ESX server itself as a compute node? If the former, VMware supports 'hot add' of CPUs, but it's been documented to cause issues with libvirtd, which might be contributing to your problem. Add/modify vcpu.hotadd = "FALSE" in the VMX for your compute node if this is the case.

    If you can check these things out and report back, we might be able to move you along. Good luck!
    Hey,

    We are running on Intel processors.

    When I executed the iaas-describe-nodes command, the storage nodes where listed, but not the compute nodes.
    Today, I executed the command again, but without success.

    Seems that my webservices aren't running anymore. http://webservice:5678/query --> unable to connect.
    Tried the ./re-run.sh script but no success.
    Any idea how to resolve? (I think it overlaps a bit with the latest post on the forum)

    To answer completely to your response,
    I'm using the ESX server itself as the compute node. ISCP installed a vmware_mvm machine to communicate with the host (I guess?)
    I verified the vmx file also and I didn't found the vcpu.hotadd = "FALSE".
    If not written, it the default value "FALSE"?
  • SystemAdmin
    SystemAdmin
    92 Posts

    Re: Storage / compute nodes don't show up in admin console.

    ‏2012-06-05T14:09:46Z  
    Hey,

    We are running on Intel processors.

    When I executed the iaas-describe-nodes command, the storage nodes where listed, but not the compute nodes.
    Today, I executed the command again, but without success.

    Seems that my webservices aren't running anymore. http://webservice:5678/query --> unable to connect.
    Tried the ./re-run.sh script but no success.
    Any idea how to resolve? (I think it overlaps a bit with the latest post on the forum)

    To answer completely to your response,
    I'm using the ESX server itself as the compute node. ISCP installed a vmware_mvm machine to communicate with the host (I guess?)
    I verified the vmx file also and I didn't found the vcpu.hotadd = "FALSE".
    If not written, it the default value "FALSE"?
    Hey, little update.

    I restarted all the services and my webservice is back up and running, but still no compute nodes showing.
    Below you'll find a print of the situation. --> Hyper manager not found.

    **************************************************
    REASON: CTGHL304E: Hyper manager not found

    0 total hyper nodes
    jid | cpu_load | ipaddress | is_manager | image_available (MB) | image_used (MB) | image_capacity (%) | volume_available (MB) | volume_used (MB) | volume_capacity (%) | version | Start Time of Request

    storage.storage-1 | 0.17 | 10.5.17.10 | y | 132795 | 4479 | 3 | 132795 | 4479 | 3 | 2.1.0.0-201205022236 | Tue Jun 05 15:24:21 +0200 2012
    storage.storage-2 | 0.07 | 10.5.17.25 | n | 126820 | 20851 | 14 | 126820 | 20851 | 14 | 2.1.0.0-201205022236 | Tue Jun 05 15:24:38 +0200 2012

    2 total storage nodes

    Transaction completed.
    **************************************************

    Next question:
    Is there a difference in command's between ESXi4.1 and 5.0?
    I'm facing the following error in the log on my compute node:

    **************************************************
    timeout: failed to run command `vmware-cmd': No such file or directory
    **************************************************

    Could this be due to the fact that I'm working on ESXi5?

    Thanks in advance for the response.
  • gengjh
    gengjh
    10 Posts

    Re: Storage / compute nodes don't show up in admin console.

    ‏2012-06-05T14:52:38Z  
    Hey, little update.

    I restarted all the services and my webservice is back up and running, but still no compute nodes showing.
    Below you'll find a print of the situation. --> Hyper manager not found.

    **************************************************
    REASON: CTGHL304E: Hyper manager not found

    0 total hyper nodes
    jid | cpu_load | ipaddress | is_manager | image_available (MB) | image_used (MB) | image_capacity (%) | volume_available (MB) | volume_used (MB) | volume_capacity (%) | version | Start Time of Request

    storage.storage-1 | 0.17 | 10.5.17.10 | y | 132795 | 4479 | 3 | 132795 | 4479 | 3 | 2.1.0.0-201205022236 | Tue Jun 05 15:24:21 +0200 2012
    storage.storage-2 | 0.07 | 10.5.17.25 | n | 126820 | 20851 | 14 | 126820 | 20851 | 14 | 2.1.0.0-201205022236 | Tue Jun 05 15:24:38 +0200 2012

    2 total storage nodes

    Transaction completed.
    **************************************************

    Next question:
    Is there a difference in command's between ESXi4.1 and 5.0?
    I'm facing the following error in the log on my compute node:

    **************************************************
    timeout: failed to run command `vmware-cmd': No such file or directory
    **************************************************

    Could this be due to the fact that I'm working on ESXi5?

    Thanks in advance for the response.
    Answer your first question:

    REASON: CTGHL304E: Hyper manager not found
    Which means your hyper bot is unreachable, you need logon your compute node and re-run the hyper bot process.
  • SystemAdmin
    SystemAdmin
    92 Posts

    Re: Storage / compute nodes don't show up in admin console.

    ‏2012-06-06T08:18:29Z  
    • gengjh
    • ‏2012-06-05T14:52:38Z
    Answer your first question:

    REASON: CTGHL304E: Hyper manager not found
    Which means your hyper bot is unreachable, you need logon your compute node and re-run the hyper bot process.
    Hey,

    Rerunning the script ./re-run.sh doesn't solve the problem.
    Tried that already a couple of times.

    I'm also facing the following problem:

    ./vmware_hyper/vmware_vcenter_adapter.rb:38:in `initialize': Could not initialize vmware adapter -- error: (RuntimeError)

    In the hyper_bot.conf you have the following parameter: "iscsi_adapter":"vmhba1".
    We don't work with iSCSI adapters but with FCoE adapters.
    Although we also use the name "vmhbaxx".

    Will this prevent the compute node from working?
  • gengjh
    gengjh
    10 Posts

    Re: Storage / compute nodes don't show up in admin console.

    ‏2012-06-06T09:19:16Z  
    Hey,

    Rerunning the script ./re-run.sh doesn't solve the problem.
    Tried that already a couple of times.

    I'm also facing the following problem:

    ./vmware_hyper/vmware_vcenter_adapter.rb:38:in `initialize': Could not initialize vmware adapter -- error: (RuntimeError)

    In the hyper_bot.conf you have the following parameter: "iscsi_adapter":"vmhba1".
    We don't work with iSCSI adapters but with FCoE adapters.
    Although we also use the name "vmhbaxx".

    Will this prevent the compute node from working?
    Please attach the log files (nohup_hyper.out and *.log) which under the /var/log/iaas on your compute node to let us check.
  • SystemAdmin
    SystemAdmin
    92 Posts

    Re: Storage / compute nodes don't show up in admin console.

    ‏2012-06-06T09:42:47Z  
    • gengjh
    • ‏2012-06-06T09:19:16Z
    Please attach the log files (nohup_hyper.out and *.log) which under the /var/log/iaas on your compute node to let us check.
    In attachment you'll find the log file of the compute node.
    I combined 3 log files in one PDF.

    Thanks for looking at them!

    And also still no nodes are shown in the admin GUI after restarting the services and verified that the nodes exist with ./iaas-describe-node script.
    They are shown in the output of the command like in my earlier post.

    Attachments

  • lzy.dev
    lzy.dev
    1 Post

    Re: Storage / compute nodes don't show up in admin console.

    ‏2012-06-06T09:45:35Z  
    Hey,

    Rerunning the script ./re-run.sh doesn't solve the problem.
    Tried that already a couple of times.

    I'm also facing the following problem:

    ./vmware_hyper/vmware_vcenter_adapter.rb:38:in `initialize': Could not initialize vmware adapter -- error: (RuntimeError)

    In the hyper_bot.conf you have the following parameter: "iscsi_adapter":"vmhba1".
    We don't work with iSCSI adapters but with FCoE adapters.
    Although we also use the name "vmhbaxx".

    Will this prevent the compute node from working?
    1. The "iscsi_adapter" option in hyper_bot.conf has been discarded. And the value is just for software iscsi adapter on esxi.

    2. Under current ISCP implementation, the instance capture function on ESXi 5 is not support yet.

    3. We need log, please post them as gengjh side.
  • bjyangf
    bjyangf
    19 Posts

    Re: Storage / compute nodes don't show up in admin console.

    ‏2012-06-06T09:46:13Z  
    Hey,

    Rerunning the script ./re-run.sh doesn't solve the problem.
    Tried that already a couple of times.

    I'm also facing the following problem:

    ./vmware_hyper/vmware_vcenter_adapter.rb:38:in `initialize': Could not initialize vmware adapter -- error: (RuntimeError)

    In the hyper_bot.conf you have the following parameter: "iscsi_adapter":"vmhba1".
    We don't work with iSCSI adapters but with FCoE adapters.
    Although we also use the name "vmhbaxx".

    Will this prevent the compute node from working?
    Please also attach /tmp/startup-<date and time>.log files here so that we can understand what happened during the setup process.
  • SystemAdmin
    SystemAdmin
    92 Posts

    Re: Storage / compute nodes don't show up in admin console.

    ‏2012-06-06T09:59:57Z  
    • bjyangf
    • ‏2012-06-06T09:46:13Z
    Please also attach /tmp/startup-<date and time>.log files here so that we can understand what happened during the setup process.
    In attachment you'll find the startup log.

    While installing the vmware-mvm, the scripts install 3 network devices.
    Can you explain the purpose of these devices please?
    They are called mgm, iaas and external by default.

    Thanks!
  • SystemAdmin
    SystemAdmin
    92 Posts

    Re: Storage / compute nodes don't show up in admin console.

    ‏2012-06-20T11:34:04Z  
    In attachment you'll find the startup log.

    While installing the vmware-mvm, the scripts install 3 network devices.
    Can you explain the purpose of these devices please?
    They are called mgm, iaas and external by default.

    Thanks!
    Any update on this issue?
    Thanks in advance.