Topic
  • 2 replies
  • Latest Post - ‏2012-05-22T22:58:41Z by k_jan
DJSchaefer
DJSchaefer
17 Posts

Pinned topic SCP Adminconsole gives wrong information on cloud infrastructure usage?

‏2012-05-22T11:33:24Z |
Hi guys,

I have another question around the admin-console of SCP.

On the page "Cloud Infrastructure" I can see the used storage on the Storage Nodes. But these values differ very much from the actual available storage on the harddisks of the storage-nodes. Please see the screenshot and compare the output from "df -h" on the system versus the output of the SCP Adminconsole.

I have also done the integration of SmartCloud Monitoring and SCP (from the ISM library) and on the Service Health for SCP navigator in the Tivoli Enterprise Portal of SCM it also shows the same wrong values as the admin-console. So probably the storage-bot somehow calculates wrong values for its available storage?

If it's just for the UI, it wouldn't be that big of a problem, but sometimes we are also unable to launch new persistent VM's due to lack of resources - although there is enough disk space left on both storage nodes.

Can anybody explain how the used and available storage is calculated for SCP and why it might get wrong?

Regards, Daniel
Updated on 2012-05-22T22:58:41Z at 2012-05-22T22:58:41Z by k_jan
  • k_jan
    k_jan
    2 Posts

    Re: SCP Adminconsole gives wrong information on cloud infrastructure usage?

    ‏2012-05-22T18:44:21Z  
    hi,

    im on the same system like daniel.
    the strage thing is that it sometimes shows u negative space avaiable like on my attachment
    but on the storage nodes there is enough space free

    when it shows up negative space, that also prevents new vms being started as persistend
    and the process deletes the vm without and response in the webgui
    the log comes up with the following:
    E, 2012-05-22T07:24:11.496608 #10784 ERROR -- kvm_rack1.v00145ebc2d9a: hyper.rb::`prepare_run_new_instance'::#<Thread:0x7f84f099ba70> (kvm_rack1.v00145ebc2d9a.51419.u49844) FAILED to launch, error:
    E, 2012-05-22T07:24:11.497630 #10784 ERROR -- kvm_rack1.v00145ebc2d9a: hyper.rb::`start_launch_worker'::#<Thread:0x7f84f099ba70> Error in prepare the instance {"error_parameters"=>, "error_code"=>"SpaceIsNotEnough", "req_id"=>"09218", "error"=>"CTGHL0075E: Storage bot does not have enough free space, it needs 40960MB but it has only -2730MB\""}
    I, 2012-05-22T07:24:29.042276 #10784 INFO -- kvm_rack1.v00145ebc2d9a: regular_bot.rb::`sync_instance_with_fe_db'::#<Thread:0x7f84f0980d10> kvm_rack1.v00145ebc2d9a.51419.u49844 has state 'failed_to_launch', clearing everything associated with it from front-end-db
    I, 2012-05-22T07:24:29.042645 #10784 INFO -- kvm_rack1.v00145ebc2d9a: regular_bot.rb::`sync_instance_with_fe_db'::#<Thread:0x7f84f0980d10> clearing persistent instance's volume status
    E, 2012-05-22T07:24:29.043102 #10784 ERROR -- kvm_rack1.v00145ebc2d9a: data_store_volume.rb::`get_volume'::#<Thread:0x7f84f0980d10> Error {"err"=>"called get with row nil."}
  • k_jan
    k_jan
    2 Posts

    Re: SCP Adminconsole gives wrong information on cloud infrastructure usage?

    ‏2012-05-22T22:58:41Z  
    • k_jan
    • ‏2012-05-22T18:44:21Z
    hi,

    im on the same system like daniel.
    the strage thing is that it sometimes shows u negative space avaiable like on my attachment
    but on the storage nodes there is enough space free

    when it shows up negative space, that also prevents new vms being started as persistend
    and the process deletes the vm without and response in the webgui
    the log comes up with the following:
    E, 2012-05-22T07:24:11.496608 #10784 ERROR -- kvm_rack1.v00145ebc2d9a: hyper.rb::`prepare_run_new_instance'::#<Thread:0x7f84f099ba70> (kvm_rack1.v00145ebc2d9a.51419.u49844) FAILED to launch, error:
    E, 2012-05-22T07:24:11.497630 #10784 ERROR -- kvm_rack1.v00145ebc2d9a: hyper.rb::`start_launch_worker'::#<Thread:0x7f84f099ba70> Error in prepare the instance {"error_parameters"=>, "error_code"=>"SpaceIsNotEnough", "req_id"=>"09218", "error"=>"CTGHL0075E: Storage bot does not have enough free space, it needs 40960MB but it has only -2730MB\""}
    I, 2012-05-22T07:24:29.042276 #10784 INFO -- kvm_rack1.v00145ebc2d9a: regular_bot.rb::`sync_instance_with_fe_db'::#<Thread:0x7f84f0980d10> kvm_rack1.v00145ebc2d9a.51419.u49844 has state 'failed_to_launch', clearing everything associated with it from front-end-db
    I, 2012-05-22T07:24:29.042645 #10784 INFO -- kvm_rack1.v00145ebc2d9a: regular_bot.rb::`sync_instance_with_fe_db'::#<Thread:0x7f84f0980d10> clearing persistent instance's volume status
    E, 2012-05-22T07:24:29.043102 #10784 ERROR -- kvm_rack1.v00145ebc2d9a: data_store_volume.rb::`get_volume'::#<Thread:0x7f84f0980d10> Error {"err"=>"called get with row nil."}
    also it seems that persistend vms cant be started again when theyre shutdown once
    maybe this is about it:

    I, 2012-05-23T00:43:43.588721 #10784 INFO -- kvm_rack1.v00145ebc2d9a: garbage_collector.rb::`file_in_use?'::#<Thread:0x7f84f099ac38> (GC) Checking if vol-43701-u49844 is used
    E, 2012-05-23T00:43:43.607299 #10784 ERROR -- kvm_rack1.v00145ebc2d9a: garbage_collector.rb::`cleanup_unused_storage_resources'::#<Thread:0x7f84f099ac38>
    NoMethodError: undefined method `[]' for nil:NilClass
    ./libvirt_hyper/../hyper/garbage_collector.rb:76:in `file_in_use?'
    ./libvirt_hyper/../hyper/garbage_collector.rb:73:in `select'
    ./libvirt_hyper/../hyper/garbage_collector.rb:73:in `file_in_use?'
    ./libvirt_hyper/../hyper/garbage_collector.rb:55:in `cleanup_unused_storage_resources'
    ./libvirt_hyper/../hyper/garbage_collector.rb:53:in `each'
    ./libvirt_hyper/../hyper/garbage_collector.rb:53:in `cleanup_unused_storage_resources'
    ./libvirt_hyper/../hyper/garbage_collector.rb:38:in `garbage_collector'
    ./libvirt_hyper/../hyper/garbage_collector.rb:22:in `start_gc'
    ./../../lib/storage_client/drivers/iscsi/../../../utilities/utils.rb:123:in `threadloop'
    ./../../lib/storage_client/drivers/iscsi/../../../utilities/utils.rb:117:in `loop'
    ./../../lib/storage_client/drivers/iscsi/../../../utilities/utils.rb:117:in `threadloop'
    ./../../lib/messaging/http_msg/trace/http_thread.rb:26:in `initialize'
    ./../../lib/messaging/http_msg/trace/http_thread.rb:23:in `_initialize'
    ./../../lib/messaging/http_msg/trace/http_thread.rb:23:in `initialize'
    ./../../lib/storage_client/drivers/iscsi/../../../utilities/utils.rb:115:in `new'
    ./../../lib/storage_client/drivers/iscsi/../../../utilities/utils.rb:115:in `threadloop'
    ./libvirt_hyper/../hyper/garbage_collector.rb:21:in `start_gc'
    ./libvirt_hyper/libvirt_hyper.rb:94:in `initialize'
    ./regular_bot.rb:99:in `new'
    ./regular_bot.rb:99:in `initialize'
    ./vm_hyper_bot.rb:41:in `new'
    ./vm_hyper_bot.rb:41:in `start_bot'
    ./vm_hyper_bot.rb:40:in `fork'
    ./vm_hyper_bot.rb:40:in `start_bot'
    ./vm_hyper_bot.rb:117:in `restart'
    ./vm_hyper_bot.rb:122