Topic
  • 3 replies
  • Latest Post - ‏2012-12-23T02:41:50Z by SystemAdmin
VincenzoVagnoni
VincenzoVagnoni
112 Posts

Pinned topic very high load of mmfsd with mmapplypolicy and list scan

‏2012-12-21T23:33:39Z |
when applying a list policy using the command:

mmapplypolicy gpfs_lhcb -P yamssListPolicy -B 100000 -N nodeA,nodeB -M YAMSSDEVICENAME=gpfs_lhcb -s /var/tmp

where yamssListPolicy is:

/* */
define(EPOCH_ATIME,INT((DAYS(ACCESS_TIME)-DAYS('1970-01-01'))*86400+HOUR(ACCESS_TIME)*3600+MINUTE(ACCESS_TIME)*60+SECOND(ACCESS_TIME)))
define(EPOCH_CTIME,INT((DAYS(CHANGE_TIME)-DAYS('1970-01-01'))*86400+HOUR(CHANGE_TIME)*3600+MINUTE(CHANGE_TIME)*60+SECOND(CHANGE_TIME)))
define(EPOCH_MTIME,INT((DAYS(MODIFICATION_TIME)-DAYS('1970-01-01'))*86400+HOUR(MODIFICATION_TIME)*3600+MINUTE(MODIFICATION_TIME)*60+SECOND(MODIFICATION_TIME)))

RULE EXTERNAL LIST 'LIST TAPE FILES' EXEC '/usr/local/yamss/bin/yamssProcessList' OPTS 'tape YAMSSDEVICENAME'
RULE 'list tape files' LIST 'LIST TAPE FILES' SHOW(VARCHAR(FILE_SIZE)||' '||VARCHAR(KB_ALLOCATED)) WHERE MISC_ATTRIBUTES LIKE '%M%'

RULE EXTERNAL LIST 'LIST ALL FILES' EXEC '/usr/local/yamss/bin/yamssProcessList' OPTS 'all YAMSSDEVICENAME'

RULE 'list all files' LIST 'LIST ALL FILES'
SHOW(VARCHAR(FILE_SIZE)||' '||
VARCHAR(KB_ALLOCATED)||' '||
VARCHAR(EPOCH_ATIME)||' '||
VARCHAR(EPOCH_CTIME)||' '||
VARCHAR(EPOCH_MTIME)||' '||
VARCHAR(FILESET_NAME)||' '||
VARCHAR(POOL_NAME)||' '||
VARCHAR(USER_ID)||' '||
VARCHAR(GROUP_ID)||' '||
VARCHAR(MISC_ATTRIBUTES))
/* */

I'm getting a very high CPU load of mmfsd, i.e. about 400%, and the machine where mmapplypolicy is issued is completely frozen for several minutes until the first phase of the scan ends and the CPU load goes down. There is plenty of memory available. Can you tell me what's wrong with this policy? GPFS is 3.4.0-17 on kernel 2.6.18-194.17.1.el5 x86_64

I did not notice this behaviour in the past with GPFS 3.3 series.
Updated on 2012-12-23T02:41:50Z at 2012-12-23T02:41:50Z by SystemAdmin
  • FredStockatIBM
    FredStockatIBM
    52 Posts

    Re: very high load of mmfsd with mmapplypolicy and list scan

    ‏2012-12-22T15:35:14Z  
    Do you have information as to the time it took for this policy to run on GPFS 3.3 versus GPFS 3.4? And presumably GPFS 3.3 was run on the identical hardware configuration but please confirm that is true.
  • VincenzoVagnoni
    VincenzoVagnoni
    112 Posts

    Re: very high load of mmfsd with mmapplypolicy and list scan

    ‏2012-12-22T15:53:17Z  
    running on 3.3 takes much longer. I have seen that tweaking -n option down from the default (24) value the load of mmfsd goes down and it takes longer. So I guess that if we want a faster scan we need to use HW with more CPU cores
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: very high load of mmfsd with mmapplypolicy and list scan

    ‏2012-12-23T02:41:50Z  
    running on 3.3 takes much longer. I have seen that tweaking -n option down from the default (24) value the load of mmfsd goes down and it takes longer. So I guess that if we want a faster scan we need to use HW with more CPU cores
    Well, that was the point of the question. If 3.4 is "burning" more CPU cycles, perhaps it's getting more scanning done per second and finishing faster.

    Be that as it may, we are very aware of these kinds of problems and are hard at work on a much better way to run "maintenance" jobs like policy and restripe, in a way that will have minimal impact on other jobs/processes/tasks that are using the file system.