Topic
  • 8 replies
  • Latest Post - ‏2012-05-22T09:13:22Z by DJSchaefer
DJSchaefer
DJSchaefer
17 Posts

Pinned topic How can I delete persistent vm's that are in status "stopped"?

‏2012-05-21T11:24:42Z |
Hi guys,

I ran into another thing and it would be great if you could give me a hint into the right direction.

My users have started a couple of persistent VM's. Now we want to remove those VM's again. We tried various combinations of "stop" and "remove" via the GUI, but although we get a message like "operation successful", those instances stick in the list of implemented VM's with the status "stopped". Is there a trick how to finally remove those persistent images from the environment?

Deleting volumes or non-persistent VM's works fine, but with persistent VM's there seems to be something else we have to consider?

Regards, Daniel
Updated on 2012-05-22T09:13:22Z at 2012-05-22T09:13:22Z by DJSchaefer
  • rossella
    rossella
    58 Posts

    Re: How can I delete persistent vm's that are in status "stopped"?

    ‏2012-05-21T16:19:36Z  
    Hi!

    Actually once you stopped the image, what you have to do to remove it is to select it in the webconsole and select "Remove". The removal should be immediate (at least it is in my environment ;-) )

    What you can do is to check, on the storage node where the volume corresponding to the persistent VM is located, the log files inside /var/log/iaas/<name of your storage node>.log
    In case of successful deletion of the volume you can find a line like
    I, <some timing information here> INFO -- storage.<name of the storage node>: http_messenger.rb::'unmark_deleted'::#<Thread:<thread number> unmarking <name of the volume> as deleted

    In addition you may want to check if the volume has been actually removed from the storage node (usually undser /iaas/iaas-store/files)
  • rossella
    rossella
    58 Posts

    Re: How can I delete persistent vm's that are in status "stopped"?

    ‏2012-05-21T16:20:38Z  
    Hi!

    Actually once you stopped the image, what you have to do to remove it is to select it in the webconsole and select "Remove". The removal should be immediate (at least it is in my environment ;-) )

    What you can do is to check, on the storage node where the volume corresponding to the persistent VM is located, the log files inside /var/log/iaas/<name of your storage node>.log
    In case of successful deletion of the volume you can find a line like
    I, <some timing information here> INFO -- storage.<name of the storage node>: http_messenger.rb::'unmark_deleted'::#<Thread:<thread number> unmarking <name of the volume> as deleted

    In addition you may want to check if the volume has been actually removed from the storage node (usually undser /iaas/iaas-store/files)
  • DJSchaefer
    DJSchaefer
    17 Posts

    Re: How can I delete persistent vm's that are in status "stopped"?

    ‏2012-05-21T18:09:20Z  
    • rossella
    • ‏2012-05-21T16:20:38Z
    Hi!

    Actually once you stopped the image, what you have to do to remove it is to select it in the webconsole and select "Remove". The removal should be immediate (at least it is in my environment ;-) )

    What you can do is to check, on the storage node where the volume corresponding to the persistent VM is located, the log files inside /var/log/iaas/<name of your storage node>.log
    In case of successful deletion of the volume you can find a line like
    I, <some timing information here> INFO -- storage.<name of the storage node>: http_messenger.rb::'unmark_deleted'::#<Thread:<thread number> unmarking <name of the volume> as deleted

    In addition you may want to check if the volume has been actually removed from the storage node (usually undser /iaas/iaas-store/files)
    Hi Rosella,

    what you describe is exactly what I would have expected it to work like.
    But in my environment it doesn't...

    When I select to remove the stopped image, it says either "operation successful" (but it doesn't change anything, it is still listed in the webconsole and is neither deleted from the storage nodes) or says that the "volume vol-... could not be found" (and also does nothing, although the volumes are present on the storage nodes).

    Is there something like a "reset button" where I can have SCP check all its images, instances and volumes whether they still exist and if not, they are deleted from GUI and the storage-nodes?
  • rossella
    rossella
    58 Posts

    Re: How can I delete persistent vm's that are in status "stopped"?

    ‏2012-05-22T07:09:40Z  
    Hi Rosella,

    what you describe is exactly what I would have expected it to work like.
    But in my environment it doesn't...

    When I select to remove the stopped image, it says either "operation successful" (but it doesn't change anything, it is still listed in the webconsole and is neither deleted from the storage nodes) or says that the "volume vol-... could not be found" (and also does nothing, although the volumes are present on the storage nodes).

    Is there something like a "reset button" where I can have SCP check all its images, instances and volumes whether they still exist and if not, they are deleted from GUI and the storage-nodes?
    No, unluckily there is not such an option unless you want to remove rows in the HBASE table, thing that I would not suggest to do. Did you had a look at the logs? You should have the reason of the failure there
  • SystemAdmin
    SystemAdmin
    92 Posts

    Re: How can I delete persistent vm's that are in status "stopped"?

    ‏2012-05-22T07:21:12Z  
    Hi Daniel, I had the same issue as you, but I was able to remove the stopped persistent image after ticking the box "Ignore errors on delete" while removing.
  • DJSchaefer
    DJSchaefer
    17 Posts

    Re: How can I delete persistent vm's that are in status "stopped"?

    ‏2012-05-22T08:37:23Z  
    Hi Daniel, I had the same issue as you, but I was able to remove the stopped persistent image after ticking the box "Ignore errors on delete" while removing.
    Hi Andrew,

    I don't see that box. Maybe it came in with a later beta drop than mine?
    Can I do a removal via the iaas-cli? I don't see a script for that?
    There is a lot of stuff in the logs (around scsi targets that seem to be still in the system, but are not present on disk??), but I don't see anything related to the deletion of images.

    I have included the /var/log/iaas from both storage nodes and the compute node in the attachment. Maybe one of you experts can find anything in here?

    Cheers, Daniel

    Attachments

  • rossella
    rossella
    58 Posts

    Re: How can I delete persistent vm's that are in status "stopped"?

    ‏2012-05-22T08:57:02Z  
    Hi Andrew,

    I don't see that box. Maybe it came in with a later beta drop than mine?
    Can I do a removal via the iaas-cli? I don't see a script for that?
    There is a lot of stuff in the logs (around scsi targets that seem to be still in the system, but are not present on disk??), but I don't see anything related to the deletion of images.

    I have included the /var/log/iaas from both storage nodes and the compute node in the attachment. Maybe one of you experts can find anything in here?

    Cheers, Daniel
    Hey, it seems you have already the volumes marked as deleted:

    I, 2012-05-22T10:33:54.598747 #23986 INFO -- storage.storage-2: local_file_store.rb::`mark_deleted'::#<Thread:0x7fbc75d374d8> marking vol-27405-u49844 as deleted
    I, 2012-05-22T10:33:54.601104 #23986 INFO -- storage.storage-2: local_file_store.rb::`mark_deleted'::#<Thread:0x7fbc75d374d8> vol-27405-u49844 is already marked as deleted
    I, 2012-05-22T10:33:54.601480 #23986 INFO -- storage.storage-2: http_base_messenger.rb::`do_json_cmd'::#<Thread:0x7fbc75d374d8> reply from=storage.manager http://172.17.42.112:8881/json_msg to=http://storage.storage-2 subject=31532 body= is {"ok"=>"ok"}

    Could you check that vol-27405-u49844 is no longer present on storage-2? Could you also check it is only available on storage-2...An interesting test I would do is to remove it from storage-2 by hand and see what the UI says at that point (of course do a backup before)
  • DJSchaefer
    DJSchaefer
    17 Posts

    Re: How can I delete persistent vm's that are in status "stopped"?

    ‏2012-05-22T09:13:22Z  
    • rossella
    • ‏2012-05-22T08:57:02Z
    Hey, it seems you have already the volumes marked as deleted:

    I, 2012-05-22T10:33:54.598747 #23986 INFO -- storage.storage-2: local_file_store.rb::`mark_deleted'::#<Thread:0x7fbc75d374d8> marking vol-27405-u49844 as deleted
    I, 2012-05-22T10:33:54.601104 #23986 INFO -- storage.storage-2: local_file_store.rb::`mark_deleted'::#<Thread:0x7fbc75d374d8> vol-27405-u49844 is already marked as deleted
    I, 2012-05-22T10:33:54.601480 #23986 INFO -- storage.storage-2: http_base_messenger.rb::`do_json_cmd'::#<Thread:0x7fbc75d374d8> reply from=storage.manager http://172.17.42.112:8881/json_msg to=http://storage.storage-2 subject=31532 body= is {"ok"=>"ok"}

    Could you check that vol-27405-u49844 is no longer present on storage-2? Could you also check it is only available on storage-2...An interesting test I would do is to remove it from storage-2 by hand and see what the UI says at that point (of course do a backup before)
    Hi Rosella,

    that volume vol-27405-u49844 is only present on storage-2 and not available on storage-1.

    I have now moved it from /iaas/iaas-store/files to /iaas/iaas-store so that it's no longer available.

    I can see the following message in the log:
    W, 2012-05-22T11:08:16.875600 #27861 WARN -- storage.storage-2: local_storage_pool.rb::`gc'::#<Thread:0x7ffa0becc608> GC:: vol-27405-u49844 is not existed but target existed , release target
    I, 2012-05-22T11:08:16.938836 #27861 INFO -- storage.storage-2: iscsi_tgt_targets_register.rb::`deregister_target'::#<Thread:0x7ffa0becc608> exit
    I, 2012-05-22T11:08:21.220808 #27861 INFO -- storage.storage-2: iaas_capture_job_manager.rb::`maintain_regular_job'::#<Thread:0x7ffa0bf0af20> no jobs need to handle, skip this round

    I have waited about 3 minutes and now it is also gone in the webconsole UI.
    Very nice! I will proceed with the other images in the same way.

    Nevertheless, it would be better if that would also work via the UI ;)

    Thanks rosella!