IC SunsetThe developerWorks Connections platform will be sunset on December 31, 2019. On January 1, 2020, this forum will no longer be available. More details available on our FAQ.
Topic
  • 5 replies
  • Latest Post - ‏2019-07-11T05:28:49Z by Jame5.H
kloby
kloby
3 Posts

Pinned topic error 565 Disk full

‏2014-07-14T13:07:20Z | v3700

Hi,

In service mode i got error on 1 canister "565 Disk full: /Var", Whats wrong? Why I dont have connection to this canister if disk is failed ?

Event ID 073565
Event ID Text Node internal disk is failing
Error Code 1032
Error Code Text Canister fault type 1
Status Alert
Fixed No
Auto Fixed No
Notification Type Error
Sense data version 0x01
Node error 565
Node error data Disk full: /var

 

IBM storewize V3700 (2072-24C) mashine ver. 7.3.0.2

  • chriscanto
    chriscanto
    292 Posts

    Re: error 565 Disk full

    ‏2014-07-14T15:58:14Z  

    I'd recommend you contact IBM Support to ask for help.

    I would expect they direct you to use the service assistant GUI (/service) to reinstall the software on that node.

    I would also hope they ask you for a support package from that node, again using the service assistant GUI, so that they can debug the root cause :-)

  • kloby
    kloby
    3 Posts

    Re: error 565 Disk full

    ‏2014-07-15T11:11:32Z  

    I'd recommend you contact IBM Support to ask for help.

    I would expect they direct you to use the service assistant GUI (/service) to reinstall the software on that node.

    I would also hope they ask you for a support package from that node, again using the service assistant GUI, so that they can debug the root cause :-)

    Thank you for answer,

    I tried reinstall machine code but than failed node didnt boot . After that I reboot working node and than error was cleared and everything is working now.

  • Jame5.H
    Jame5.H
    3 Posts

    Re: error 565 Disk full

    ‏2019-07-09T16:07:50Z  

    I realise this is an old post, but I just had this happen to me on a 3 x IO Group SV1 cluster running v8.2.1.2 code and two nodes went into service at the same time in two different IO groups.

     

    The issue is that there will be logs and dumps not cleaned up on the node and the way IBM check the disk are working I believe is to write a file which if it can't do, flags the disk as faulty and puts the node into service. Unfortunately, once the node is in service, you will need to putty directly to the service IP of that node using superuser to fix it.

    Once you get terminal access to the node thats in service, do the following:

    IBM_2145:superuser>satask snap -clean
    IBM_2145:superuser>satask stopnode -reboot

    Once the node readds itself back online in the cluster you can proceed to clean up additional files on the node using the following command from the cluster cli:

    IBM_2145:administrator>cleardumps -prefix /dumps <nodeid>

     

    So moving forward, best to use the 'svctask cleardumps -prefix /dumps <nodeid>' command before capturing fresh snaps to upload to IBM. That way you wont run into the risk of filling the disk by generating a snap and other than that stay on a current code level which usually has software enhancements to automatically try and tidy up old files.

  • fincherjc
    fincherjc
    6 Posts

    Re: error 565 Disk full

    ‏2019-07-11T02:38:21Z  
    • Jame5.H
    • ‏2019-07-09T16:07:50Z

    I realise this is an old post, but I just had this happen to me on a 3 x IO Group SV1 cluster running v8.2.1.2 code and two nodes went into service at the same time in two different IO groups.

     

    The issue is that there will be logs and dumps not cleaned up on the node and the way IBM check the disk are working I believe is to write a file which if it can't do, flags the disk as faulty and puts the node into service. Unfortunately, once the node is in service, you will need to putty directly to the service IP of that node using superuser to fix it.

    Once you get terminal access to the node thats in service, do the following:

    IBM_2145:superuser>satask snap -clean
    IBM_2145:superuser>satask stopnode -reboot

    Once the node readds itself back online in the cluster you can proceed to clean up additional files on the node using the following command from the cluster cli:

    IBM_2145:administrator>cleardumps -prefix /dumps <nodeid>

     

    So moving forward, best to use the 'svctask cleardumps -prefix /dumps <nodeid>' command before capturing fresh snaps to upload to IBM. That way you wont run into the risk of filling the disk by generating a snap and other than that stay on a current code level which usually has software enhancements to automatically try and tidy up old files.

    Hello friends,

    I would normally advise against blanket clearing dumps before getting a new snap for support unless you know you are going to be collecting several snaps with livedumps on a daily basis. Doing a svctask cleardumps -prefix /dumps will delete a lot of valuable information that may be needed by support depending on your issue (performance metrics for example would be deleted prior to collection). Instead I might suggest just deleting old snap and livedumps by running these scripts (as superuser).

     

    svc_snap -clean && sainfo lsservicenodes -nohdr | while read id trash; do satask snap -clean $id; done

    sainfo lsservicenodes -nohdr | while read panel trash; do sainfo lsfiles $panel | grep livedump | grep -Ev 'ietd|keymgr' | while read dumps; do cleardumps -prefix /dumps/$dumps; done; done

     

    The first cleans all the old snaps. The second cleans all the old livedumps. The livedump cleaning script can be modified for full dumps, but full dumps are taken when nodes reboot (or assert) and space is automatically cleared when this happens for new full dumps. Additionally old full dumps may provide IBM support some value for root cause analysis.

    Updated on 2019-07-11T18:13:55Z at 2019-07-11T18:13:55Z by fincherjc
  • Jame5.H
    Jame5.H
    3 Posts

    Re: error 565 Disk full

    ‏2019-07-11T05:28:49Z  
    • fincherjc
    • ‏2019-07-11T02:38:21Z

    Hello friends,

    I would normally advise against blanket clearing dumps before getting a new snap for support unless you know you are going to be collecting several snaps with livedumps on a daily basis. Doing a svctask cleardumps -prefix /dumps will delete a lot of valuable information that may be needed by support depending on your issue (performance metrics for example would be deleted prior to collection). Instead I might suggest just deleting old snap and livedumps by running these scripts (as superuser).

     

    svc_snap -clean && sainfo lsservicenodes -nohdr | while read id trash; do satask snap -clean $id; done

    sainfo lsservicenodes -nohdr | while read panel trash; do sainfo lsfiles $panel | grep livedump | grep -Ev 'ietd|keymgr' | while read dumps; do cleardumps -prefix /dumps/$dumps; done; done

     

    The first cleans all the old snaps. The second cleans all the old livedumps. The livedump cleaning script can be modified for full dumps, but full dumps are taken when nodes reboot (or assert) and space is automatically cleared when this happens for new full dumps. Additionally old full dumps may provide IBM support some value for root cause analysis.

    Thanks for the tips fincherjc, good points about wiping critical data with cleardumps. I'm going to include those steps in my notes.

     

    Obviously the first line you provided had a typo as 'svcinfo lsservicenodes' is invalid and should of been 'sainfo lsservicenodes' Wink

     

    I try to avoid superuser login as much as possible within our team (breaks our audit compliance), but the sainfo and satask commands can do so much more than the standard svcinfo/svctask commands sometimes!