Topic
  • 14 replies
  • Latest Post - ‏2013-01-21T09:56:52Z by ValerieR
SystemAdmin
SystemAdmin
2092 Posts

Pinned topic how to clear gpfs cache manually?

‏2012-12-11T05:49:46Z |
Which command can clear gpfs cache?
I have to shutdown and start gpfs to clear cache. this sulution is troublesome and time consuming.
Any cammand can do this?
Updated on 2013-01-21T09:56:52Z at 2013-01-21T09:56:52Z by ValerieR
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: how to clear gpfs cache manually?

    ‏2012-12-11T15:00:23Z  
    See mmfsctl xxx suspend/resume.

    That will flush caches to disk. I don't know offhand if it will also clear the various caches.
    What is it you're trying to accomplish and why?
  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: how to clear gpfs cache manually?

    ‏2012-12-11T21:43:26Z  
    See mmfsctl xxx suspend/resume.

    That will flush caches to disk. I don't know offhand if it will also clear the various caches.
    What is it you're trying to accomplish and why?
    mmfsctl $fsname suspend/resume will only flush the dirty data. It will not empty the cache. Unfortunately, the only way to clear the cache is to unmount/mount the filesystem. And even this will not do anything on any "manager" node that are handling tokens for the filesystem.
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: how to clear gpfs cache manually?

    ‏2012-12-12T05:55:37Z  
    See mmfsctl xxx suspend/resume.

    That will flush caches to disk. I don't know offhand if it will also clear the various caches.
    What is it you're trying to accomplish and why?
    I am doing some full-text search tests on gpfs.
    The full-text search engine has many parameters which need to configured.
    I need to test which parameters have the best performance.
    In test, I find gpfs cache will affect the program.
    when i run a test case, I must clear gpfs cache.
    otherwise,when I try a different parameters configuration, and rerun the test case again.
    it will have a better performance than previous one.

    So I must clear gpfs cache before rerun the test.
  • sberman
    sberman
    62 Posts

    Re: how to clear gpfs cache manually?

    ‏2012-12-12T13:59:22Z  
    I am doing some full-text search tests on gpfs.
    The full-text search engine has many parameters which need to configured.
    I need to test which parameters have the best performance.
    In test, I find gpfs cache will affect the program.
    when i run a test case, I must clear gpfs cache.
    otherwise,when I try a different parameters configuration, and rerun the test case again.
    it will have a better performance than previous one.

    So I must clear gpfs cache before rerun the test.
    If this is for test and performance analysis only, not a regular production operation, as Dan said already, the only way to clear all cached data is to unmount the GPFS file system. mmunmount and then mmmount it back again. You will have no cached data left over, but all access to the file system will be interrupted.
  • HajoEhlers
    HajoEhlers
    253 Posts

    Re: how to clear gpfs cache manually?

    ‏2012-12-12T15:30:42Z  
    I am doing some full-text search tests on gpfs.
    The full-text search engine has many parameters which need to configured.
    I need to test which parameters have the best performance.
    In test, I find gpfs cache will affect the program.
    when i run a test case, I must clear gpfs cache.
    otherwise,when I try a different parameters configuration, and rerun the test case again.
    it will have a better performance than previous one.

    So I must clear gpfs cache before rerun the test.
    You have some other options as well depending want you looking for.

    1 ) Clear the GPFS pagepool cache ( Metadata is still cached ) - by making it invalid.
    Note: I assume that the seqDiscardThreshold can be dynamicly changed. If not i would suggest to set the seqDiscardThreshold to the same size as the pagepool.

    
    Set  seqDiscardThreshold to the size of the pagepool Write/read a file in the size of the pagepool reset  seqDiscardThreshold to its original value   or in 
    
    case the pagepool is to large:   decrease the pagepool to a small value ( 64MB) Set  seqDiscardThreshold to the size of the pagepool Write/read a file in the size of the pagepool reset the pagepool to its original value. reset seqDiscardThreshold to its original value
    


    This should overwrite everything in the pagepool
    2) Use directo i/o to bypass GPFS cache.
    Use the mmchattr -D to set direct i/o flag for all files which shall not be cached.
    All not tested
    Cheers
    Hajo
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: how to clear gpfs cache manually?

    ‏2012-12-12T20:16:04Z  
    You have some other options as well depending want you looking for.

    1 ) Clear the GPFS pagepool cache ( Metadata is still cached ) - by making it invalid.
    Note: I assume that the seqDiscardThreshold can be dynamicly changed. If not i would suggest to set the seqDiscardThreshold to the same size as the pagepool.

    <pre class="jive-pre"> Set seqDiscardThreshold to the size of the pagepool Write/read a file in the size of the pagepool reset seqDiscardThreshold to its original value or in case the pagepool is to large: decrease the pagepool to a small value ( 64MB) Set seqDiscardThreshold to the size of the pagepool Write/read a file in the size of the pagepool reset the pagepool to its original value. reset seqDiscardThreshold to its original value </pre>

    This should overwrite everything in the pagepool
    2) Use directo i/o to bypass GPFS cache.
    Use the mmchattr -D to set direct i/o flag for all files which shall not be cached.
    All not tested
    Cheers
    Hajo
    Your safest option which is only a few seconds of interruption is to

    mmshutdown -a;
    mmstartup -a;

    mmmount xyz -a (if you don't have automount set)

    Shutting down the daemon kills the mmfsd (GPFS) process on each node, so for sure, you will be starting "clean" wrt GPFS.
    If you just unmount, then you can't be sure that there is no buffer/memory "leak" that is accumulating between tests.
    No, I'm not saying mmfsd has memory leasks, but I couldn't guaranteee 100% that it doesn't either!
  • ValerieR
    ValerieR
    11 Posts

    Re: how to clear gpfs cache manually?

    ‏2012-12-26T16:15:03Z  
    Your safest option which is only a few seconds of interruption is to

    mmshutdown -a;
    mmstartup -a;

    mmmount xyz -a (if you don't have automount set)

    Shutting down the daemon kills the mmfsd (GPFS) process on each node, so for sure, you will be starting "clean" wrt GPFS.
    If you just unmount, then you can't be sure that there is no buffer/memory "leak" that is accumulating between tests.
    No, I'm not saying mmfsd has memory leasks, but I couldn't guaranteee 100% that it doesn't either!
    Hello guys

    I am also interested by this question about flushing or clear the gpfs cache.
    My problem is in another context (production).
    For backup purpose, we use a IBM flashcopy like process to copy GPFS data on a disaster recovery/backup server, thru a Storage Array disk copy mechanism(Business Copy on XP24000 Storage Array). This work correctly except for one file, that is created just before the split of the copy (this file is used to open the Oracle database in recover mode). From time to time (not every day), this file appear to be empty (size 0) on the backup server.
    On the source (production) server, this file is not empty. It's a very little file in size (87 bytes).

    We make multiple searches to try to understand why this file was sometimes empty. One of the supposition is that this file has not been flushed from the GPFS cache before the split command.
    In the context of our customer, it's not possible to unmount the GPFS FS (Oracle RAC with SAP application), as we do online backup. So one of my idea was to use the Direct I/O attribute on this file.
    As I read before in this post, do you think it's a good idea and that it could solve my problem ?
    Does somebody already use this attribute and in what case?

    Thanks a lot for your help
    Valerie
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: how to clear gpfs cache manually?

    ‏2012-12-26T16:29:43Z  
    • ValerieR
    • ‏2012-12-26T16:15:03Z
    Hello guys

    I am also interested by this question about flushing or clear the gpfs cache.
    My problem is in another context (production).
    For backup purpose, we use a IBM flashcopy like process to copy GPFS data on a disaster recovery/backup server, thru a Storage Array disk copy mechanism(Business Copy on XP24000 Storage Array). This work correctly except for one file, that is created just before the split of the copy (this file is used to open the Oracle database in recover mode). From time to time (not every day), this file appear to be empty (size 0) on the backup server.
    On the source (production) server, this file is not empty. It's a very little file in size (87 bytes).

    We make multiple searches to try to understand why this file was sometimes empty. One of the supposition is that this file has not been flushed from the GPFS cache before the split command.
    In the context of our customer, it's not possible to unmount the GPFS FS (Oracle RAC with SAP application), as we do online backup. So one of my idea was to use the Direct I/O attribute on this file.
    As I read before in this post, do you think it's a good idea and that it could solve my problem ?
    Does somebody already use this attribute and in what case?

    Thanks a lot for your help
    Valerie
    ValerieR,
    Are you using mmfsctl fsname suspend; <flash-copy!!>; mmfsname resume;

    That should work. If not, there is a bug somewhere.

    I believe just about any other sequence you try will have a window of vulnerability.

    FYI, small files that will fit in an inode, are stored in the inode with the latest release of GPFS.
  • ValerieR
    ValerieR
    11 Posts

    Re: how to clear gpfs cache manually?

    ‏2012-12-27T08:09:43Z  
    ValerieR,
    Are you using mmfsctl fsname suspend; <flash-copy!!>; mmfsname resume;

    That should work. If not, there is a bug somewhere.

    I believe just about any other sequence you try will have a window of vulnerability.

    FYI, small files that will fit in an inode, are stored in the inode with the latest release of GPFS.
    Hello

    No, we don't use the mmfsctl suspend / resume commands at this time. It was a decision validated with IBM to not use this command, to avoid loss of application IO during this period.
    If we have a guaranty that when you do a suspend, the IO are kept somewhere in memory but not loss, we can try to use this command, but I saw anything about this in the documentation I read and i'm not sure that we don't loose any IO. What happend exactly when you use the mmfsctl suspend command ? Does it make some retry, does it generate some system errors ?

    I understand that mmfsctl suspend command will be the best solution for my problem, but if we still not want to use these commands, can we suppose that the DIO attribute could solve our problem ?

    We are at this time in release 3.3.0.20 of GPFS. As it take a long time to be able to migrate thru a new release of GPFS because of a production environment, we cannot go into a new release before the end of 2013 year !!! But we are going to prepare during the beginning of 2013 the future migration to 3.5 release.

    Thanks for your help
    Valérie
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: how to clear gpfs cache manually?

    ‏2012-12-27T16:20:25Z  
    • ValerieR
    • ‏2012-12-27T08:09:43Z
    Hello

    No, we don't use the mmfsctl suspend / resume commands at this time. It was a decision validated with IBM to not use this command, to avoid loss of application IO during this period.
    If we have a guaranty that when you do a suspend, the IO are kept somewhere in memory but not loss, we can try to use this command, but I saw anything about this in the documentation I read and i'm not sure that we don't loose any IO. What happend exactly when you use the mmfsctl suspend command ? Does it make some retry, does it generate some system errors ?

    I understand that mmfsctl suspend command will be the best solution for my problem, but if we still not want to use these commands, can we suppose that the DIO attribute could solve our problem ?

    We are at this time in release 3.3.0.20 of GPFS. As it take a long time to be able to migrate thru a new release of GPFS because of a production environment, we cannot go into a new release before the end of 2013 year !!! But we are going to prepare during the beginning of 2013 the future migration to 3.5 release.

    Thanks for your help
    Valérie
    Well that was my simple q&a....

    Now let's step back and see what you are really trying to accomplish!

    You have a GPFS filesystem, and an Oracle database, (anything else? !!)
    In theory, I presume, everything is programmed and set up so you would "survive" a sudden power outage followed by restart. Yes?

    That is usually done via a combination of techniques, including data base logging, file system logging, file system fsync calls,
    NVRAM in your disk controllers, etc, etc. In short, at any instant, the state of your system, even excluding "volatile" memory buffers, is spread over several components and subsystems!

    So, unless you've thought all that through, and can prove that your flash copy is capturing everything it needs to capture -
    you'd be fooled to think that the flash copy is sufficient for recovery purposes...

    If you really want to continue down this line, you've got to be sure that everything is going to the drives that are being flash copied, and that the flash copy is truly a complete "point-in-time" copy.

    Then you have to come up with a recovery procedure, which would probably be - a complete power-off shutdown of all relevant systems,
    then bring up your disk drive system; restore it to the point-in-time. Then bring up the filesystem, let it run through its own log recovery procedures (GPFS will do that before it declares a mmmount successful.) Then bring up the database, let it run through its log recovery, etc, etc.

    Good luck! This is not easy stuff. Don't let any sales rep type tell you differently! (IBM, Oracle or otherwise ;-) !! )
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: how to clear gpfs cache manually?

    ‏2013-01-02T05:58:12Z  
    • ValerieR
    • ‏2012-12-27T08:09:43Z
    Hello

    No, we don't use the mmfsctl suspend / resume commands at this time. It was a decision validated with IBM to not use this command, to avoid loss of application IO during this period.
    If we have a guaranty that when you do a suspend, the IO are kept somewhere in memory but not loss, we can try to use this command, but I saw anything about this in the documentation I read and i'm not sure that we don't loose any IO. What happend exactly when you use the mmfsctl suspend command ? Does it make some retry, does it generate some system errors ?

    I understand that mmfsctl suspend command will be the best solution for my problem, but if we still not want to use these commands, can we suppose that the DIO attribute could solve our problem ?

    We are at this time in release 3.3.0.20 of GPFS. As it take a long time to be able to migrate thru a new release of GPFS because of a production environment, we cannot go into a new release before the end of 2013 year !!! But we are going to prepare during the beginning of 2013 the future migration to 3.5 release.

    Thanks for your help
    Valérie
    I'm not sure what "loss of application IO" means. It appears that the fear is that after "mmfsctl suspend" is issued, some IOs could be failed or discarded. This most certainly won't happen -- a command with such semantics would be less than useless. After "mmfsctl suspend" is issued, IOs in progress are allowed to complete, while new incoming IOs are blocked until "mmfsctl resume" is issued. After all IOs already in progress complete, all data and metadata buffers are flushed to disk. Note that this does not mean that caches are cleared. The basic idea of this operation is to provide a self-consistent disk image that would be a good target for a flash copy, i.e. exactly what you're looking for.

    yuri
  • ValerieR
    ValerieR
    11 Posts

    Re: how to clear gpfs cache manually?

    ‏2013-01-18T16:03:29Z  
    I'm not sure what "loss of application IO" means. It appears that the fear is that after "mmfsctl suspend" is issued, some IOs could be failed or discarded. This most certainly won't happen -- a command with such semantics would be less than useless. After "mmfsctl suspend" is issued, IOs in progress are allowed to complete, while new incoming IOs are blocked until "mmfsctl resume" is issued. After all IOs already in progress complete, all data and metadata buffers are flushed to disk. Note that this does not mean that caches are cleared. The basic idea of this operation is to provide a self-consistent disk image that would be a good target for a flash copy, i.e. exactly what you're looking for.

    yuri
    Hello

    Thanks for all your answers.

    Concerning the mmfsctl suspend command, as I understand, no IO are losed, while in this suspend mode.
    But from the application point of view, what are the impacts ? can we see the application blocked or can we have an IO error, for example, especially if the suspend time is long ?

    Best regards
    Valerie
  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: how to clear gpfs cache manually?

    ‏2013-01-18T16:52:02Z  
    • ValerieR
    • ‏2013-01-18T16:03:29Z
    Hello

    Thanks for all your answers.

    Concerning the mmfsctl suspend command, as I understand, no IO are losed, while in this suspend mode.
    But from the application point of view, what are the impacts ? can we see the application blocked or can we have an IO error, for example, especially if the suspend time is long ?

    Best regards
    Valerie
    During the suspend all requests are just held until resume happens. There are no errors returned.
  • ValerieR
    ValerieR
    11 Posts

    Re: how to clear gpfs cache manually?

    ‏2013-01-21T09:56:52Z  
    • dlmcnabb
    • ‏2013-01-18T16:52:02Z
    During the suspend all requests are just held until resume happens. There are no errors returned.
    Hello
    Thank you very much for the answer.

    Best regards
    Valerie