Topic
8 replies Latest Post - ‏2012-05-22T09:13:22Z by DJSchaefer
DJSchaefer
DJSchaefer
17 Posts
ACCEPTED ANSWER

Pinned topic How can I delete persistent vm's that are in status "stopped"?

‏2012-05-21T11:24:42Z |
Hi guys,

I ran into another thing and it would be great if you could give me a hint into the right direction.

My users have started a couple of persistent VM's. Now we want to remove those VM's again. We tried various combinations of "stop" and "remove" via the GUI, but although we get a message like "operation successful", those instances stick in the list of implemented VM's with the status "stopped". Is there a trick how to finally remove those persistent images from the environment?

Deleting volumes or non-persistent VM's works fine, but with persistent VM's there seems to be something else we have to consider?

Regards, Daniel
Updated on 2012-05-22T09:13:22Z at 2012-05-22T09:13:22Z by DJSchaefer
  • rossella
    rossella
    58 Posts
    ACCEPTED ANSWER

    Re: How can I delete persistent vm's that are in status "stopped"?

    ‏2012-05-21T16:19:36Z  in response to DJSchaefer
    Hi!

    Actually once you stopped the image, what you have to do to remove it is to select it in the webconsole and select "Remove". The removal should be immediate (at least it is in my environment ;-) )

    What you can do is to check, on the storage node where the volume corresponding to the persistent VM is located, the log files inside /var/log/iaas/<name of your storage node>.log
    In case of successful deletion of the volume you can find a line like
    I, <some timing information here> INFO -- storage.<name of the storage node>: http_messenger.rb::'unmark_deleted'::#<Thread:<thread number> unmarking <name of the volume> as deleted

    In addition you may want to check if the volume has been actually removed from the storage node (usually undser /iaas/iaas-store/files)
  • rossella
    rossella
    58 Posts
    ACCEPTED ANSWER

    Re: How can I delete persistent vm's that are in status "stopped"?

    ‏2012-05-21T16:20:38Z  in response to DJSchaefer
    Hi!

    Actually once you stopped the image, what you have to do to remove it is to select it in the webconsole and select "Remove". The removal should be immediate (at least it is in my environment ;-) )

    What you can do is to check, on the storage node where the volume corresponding to the persistent VM is located, the log files inside /var/log/iaas/<name of your storage node>.log
    In case of successful deletion of the volume you can find a line like
    I, <some timing information here> INFO -- storage.<name of the storage node>: http_messenger.rb::'unmark_deleted'::#<Thread:<thread number> unmarking <name of the volume> as deleted

    In addition you may want to check if the volume has been actually removed from the storage node (usually undser /iaas/iaas-store/files)
    • DJSchaefer
      DJSchaefer
      17 Posts
      ACCEPTED ANSWER

      Re: How can I delete persistent vm's that are in status "stopped"?

      ‏2012-05-21T18:09:20Z  in response to rossella
      Hi Rosella,

      what you describe is exactly what I would have expected it to work like.
      But in my environment it doesn't...

      When I select to remove the stopped image, it says either "operation successful" (but it doesn't change anything, it is still listed in the webconsole and is neither deleted from the storage nodes) or says that the "volume vol-... could not be found" (and also does nothing, although the volumes are present on the storage nodes).

      Is there something like a "reset button" where I can have SCP check all its images, instances and volumes whether they still exist and if not, they are deleted from GUI and the storage-nodes?
      • rossella
        rossella
        58 Posts
        ACCEPTED ANSWER

        Re: How can I delete persistent vm's that are in status "stopped"?

        ‏2012-05-22T07:09:40Z  in response to DJSchaefer
        No, unluckily there is not such an option unless you want to remove rows in the HBASE table, thing that I would not suggest to do. Did you had a look at the logs? You should have the reason of the failure there
  • SystemAdmin
    SystemAdmin
    92 Posts
    ACCEPTED ANSWER

    Re: How can I delete persistent vm's that are in status "stopped"?

    ‏2012-05-22T07:21:12Z  in response to DJSchaefer
    Hi Daniel, I had the same issue as you, but I was able to remove the stopped persistent image after ticking the box "Ignore errors on delete" while removing.
    • DJSchaefer
      DJSchaefer
      17 Posts
      ACCEPTED ANSWER

      Re: How can I delete persistent vm's that are in status "stopped"?

      ‏2012-05-22T08:37:23Z  in response to SystemAdmin
      Hi Andrew,

      I don't see that box. Maybe it came in with a later beta drop than mine?
      Can I do a removal via the iaas-cli? I don't see a script for that?
      There is a lot of stuff in the logs (around scsi targets that seem to be still in the system, but are not present on disk??), but I don't see anything related to the deletion of images.

      I have included the /var/log/iaas from both storage nodes and the compute node in the attachment. Maybe one of you experts can find anything in here?

      Cheers, Daniel

      Attachments

      • rossella
        rossella
        58 Posts
        ACCEPTED ANSWER

        Re: How can I delete persistent vm's that are in status "stopped"?

        ‏2012-05-22T08:57:02Z  in response to DJSchaefer
        Hey, it seems you have already the volumes marked as deleted:

        I, 2012-05-22T10:33:54.598747 #23986 INFO -- storage.storage-2: local_file_store.rb::`mark_deleted'::#<Thread:0x7fbc75d374d8> marking vol-27405-u49844 as deleted
        I, 2012-05-22T10:33:54.601104 #23986 INFO -- storage.storage-2: local_file_store.rb::`mark_deleted'::#<Thread:0x7fbc75d374d8> vol-27405-u49844 is already marked as deleted
        I, 2012-05-22T10:33:54.601480 #23986 INFO -- storage.storage-2: http_base_messenger.rb::`do_json_cmd'::#<Thread:0x7fbc75d374d8> reply from=storage.manager http://172.17.42.112:8881/json_msg to=http://storage.storage-2 subject=31532 body= is {"ok"=>"ok"}

        Could you check that vol-27405-u49844 is no longer present on storage-2? Could you also check it is only available on storage-2...An interesting test I would do is to remove it from storage-2 by hand and see what the UI says at that point (of course do a backup before)
        • DJSchaefer
          DJSchaefer
          17 Posts
          ACCEPTED ANSWER

          Re: How can I delete persistent vm's that are in status "stopped"?

          ‏2012-05-22T09:13:22Z  in response to rossella
          Hi Rosella,

          that volume vol-27405-u49844 is only present on storage-2 and not available on storage-1.

          I have now moved it from /iaas/iaas-store/files to /iaas/iaas-store so that it's no longer available.

          I can see the following message in the log:
          W, 2012-05-22T11:08:16.875600 #27861 WARN -- storage.storage-2: local_storage_pool.rb::`gc'::#<Thread:0x7ffa0becc608> GC:: vol-27405-u49844 is not existed but target existed , release target
          I, 2012-05-22T11:08:16.938836 #27861 INFO -- storage.storage-2: iscsi_tgt_targets_register.rb::`deregister_target'::#<Thread:0x7ffa0becc608> exit
          I, 2012-05-22T11:08:21.220808 #27861 INFO -- storage.storage-2: iaas_capture_job_manager.rb::`maintain_regular_job'::#<Thread:0x7ffa0bf0af20> no jobs need to handle, skip this round

          I have waited about 3 minutes and now it is also gone in the webconsole UI.
          Very nice! I will proceed with the other images in the same way.

          Nevertheless, it would be better if that would also work via the UI ;)

          Thanks rosella!