Topic
  • 9 replies
  • Latest Post - ‏2012-12-21T18:57:24Z by SystemAdmin
Tucks
Tucks
78 Posts

Pinned topic SHOW(FILE_HEAT) does not elicit a value?

‏2012-12-17T12:07:36Z |
Hello

I'm playing with FILE_HEAT in 3.5.0.7 x86_64.
I have enabled fileheatperiodminutes and fileheatlosspercent.
I've randomly read the files in my testdata folder.

I have a simple policy:


RULE EXTERNAL LIST 
'genheat' EXEC 
'/mnt/gpfs/online/listproc'   RULE 
'heat1' LIST 
'genheat' WEIGHT(INODE) SHOW(FILE_HEAT) WHERE NAME LIKE 
'%'


Which generates a line per file.
However, when I use FILE_HEAT, I see no value.


4098 1002638089 0   -- /mnt/gpfs/online/testdata/150527.file 4097 69228581 0   -- /mnt/gpfs/online/testdata/150526.file 4096 769323719 0   -- /mnt/gpfs/online/testdata/150525.file


However, if I substitute FILE_HEAT for INODE, I do see a value.


RULE 
'heat1' LIST 
'genheat' WEIGHT(INODE) SHOW(INODE) WHERE NAME LIKE 
'%'


Result:


4098 1002638089 0  4098 -- /mnt/gpfs/online/testdata/150527.file 4097 69228581 0  4097 -- /mnt/gpfs/online/testdata/150526.file 4096 769323719 0  4096 -- /mnt/gpfs/online/testdata/150525.file


Qu1. Why do I not see a value for FILE_HEAT in any of my testdata files? What am I missing?
Qu2. If there is no value for FILE_HEAT, why not put 0 rather than no value at all? It makes for easier parsing.
Updated on 2012-12-21T18:57:24Z at 2012-12-21T18:57:24Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: SHOW(FILE_HEAT) does not elicit a value?

    ‏2012-12-17T17:10:18Z  
    The FILE_HEAT implementation is based on Extended Attributes which may have an SQL NULL value.

    Try this, then examine it carefully. Then RTFM. If still stumped, come back here ;-)

    
    [root@fin44 gpfs-git]# cat /ghome/makaplan/policies/fileheat-simp.policy define(DISPLAY_NULL,[CASE WHEN ($1) IS NULL THEN 
    '_NULL_' ELSE varchar($1) END])   rule fh1 external list 
    'fh' exec 
    '' rule fh2 list 
    'fh' weight(FILE_HEAT) show(
    '++' DISPLAY_NULL(FILE_HEAT) )   [root@fin44 gpfs-git]# mmapplypolicy /zzz/fat -P /ghome/makaplan/policies/fileheat-simp.policy -I defer -f /tmp/fh ... [root@fin44 gpfs-git]# cat /tmp/fh.list.fh 17670 856063417 0 ++ _NULL_ -- /zzz/fat/a20K 17671 1002494513 0 ++ +0.00000000000000E+000 -- /zzz/fat/shoot 23937 425139714 0 ++ +0.00000000000000E+000 -- /zzz/fat/m100M 23938 1010126561 0 ++ +0.00000000000000E+000 -- /zzz/fat/m50M
    
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: SHOW(FILE_HEAT) does not elicit a value?

    ‏2012-12-17T17:19:29Z  
    The FILE_HEAT implementation is based on Extended Attributes which may have an SQL NULL value.

    Try this, then examine it carefully. Then RTFM. If still stumped, come back here ;-)

    <pre class="jive-pre"> [root@fin44 gpfs-git]# cat /ghome/makaplan/policies/fileheat-simp.policy define(DISPLAY_NULL,[CASE WHEN ($1) IS NULL THEN '_NULL_' ELSE varchar($1) END]) rule fh1 external list 'fh' exec '' rule fh2 list 'fh' weight(FILE_HEAT) show( '++' DISPLAY_NULL(FILE_HEAT) ) [root@fin44 gpfs-git]# mmapplypolicy /zzz/fat -P /ghome/makaplan/policies/fileheat-simp.policy -I defer -f /tmp/fh ... [root@fin44 gpfs-git]# cat /tmp/fh.list.fh 17670 856063417 0 ++ _NULL_ -- /zzz/fat/a20K 17671 1002494513 0 ++ +0.00000000000000E+000 -- /zzz/fat/shoot 23937 425139714 0 ++ +0.00000000000000E+000 -- /zzz/fat/m100M 23938 1010126561 0 ++ +0.00000000000000E+000 -- /zzz/fat/m50M </pre>
    I think the NULL value tells you that the file has not been accessed, not at all, since FILE HEAT tracking has been enabled.

    If you'd rather see 0, just modify your DISPLAY_NULL macro.

    If you are more curious, you can study the macro expansion of your policy file:

    
    rule fh1 external list 
    'fh' exec 
    '' rule fh2 list 
    'fh' weight(computeFileHeat(CURRENT_TIMESTAMP-ACCESS_TIME,xattr(
    'gpfs.FileHeat'),KB_ALLOCATED)) show(
    '++' CASE WHEN (computeFileHeat(CURRENT_TIMESTAMP-ACCESS_TIME,xattr(
    'gpfs.FileHeat'),KB_ALLOCATED)) IS NULL THEN 
    '_NULL_' ELSE varchar(computeFileHeat(CURRENT_TIMESTAMP-ACCESS_TIME,xattr(
    'gpfs.FileHeat'),KB_ALLOCATED)) END )
    


    And notice that FILE_HEAT itself is a macro.
  • Tucks
    Tucks
    78 Posts

    Re: SHOW(FILE_HEAT) does not elicit a value?

    ‏2012-12-17T21:33:22Z  
    The FILE_HEAT implementation is based on Extended Attributes which may have an SQL NULL value.

    Try this, then examine it carefully. Then RTFM. If still stumped, come back here ;-)

    <pre class="jive-pre"> [root@fin44 gpfs-git]# cat /ghome/makaplan/policies/fileheat-simp.policy define(DISPLAY_NULL,[CASE WHEN ($1) IS NULL THEN '_NULL_' ELSE varchar($1) END]) rule fh1 external list 'fh' exec '' rule fh2 list 'fh' weight(FILE_HEAT) show( '++' DISPLAY_NULL(FILE_HEAT) ) [root@fin44 gpfs-git]# mmapplypolicy /zzz/fat -P /ghome/makaplan/policies/fileheat-simp.policy -I defer -f /tmp/fh ... [root@fin44 gpfs-git]# cat /tmp/fh.list.fh 17670 856063417 0 ++ _NULL_ -- /zzz/fat/a20K 17671 1002494513 0 ++ +0.00000000000000E+000 -- /zzz/fat/shoot 23937 425139714 0 ++ +0.00000000000000E+000 -- /zzz/fat/m100M 23938 1010126561 0 ++ +0.00000000000000E+000 -- /zzz/fat/m50M </pre>
    So.. here's a reproduction of the issue.
    I created two test files and cat the contents of one.

    As you can see, no heat value is inserted into the LIST (listproc just copies the list file to a known filename).
    I.E. the values are still NULL (thanks for the NULL pointer (bad pun..) btw. I'll be using that lots).

    I'm expecting both files to have a heat value.
    The file that was read being higher than the other.

    Note: after the mmchconfig and recycling of mmfsd, fileheatlosspercent is not present in mmlsconfig. Weird?

    
    [root@snoopy ~]# mmchconfig fileheatperiodminutes=1,fileheatlosspercent=10 mmchconfig: Command successfully completed [root@snoopy ~]# mmshutdown Mon Dec 17 20:56:42 GMT 2012: mmshutdown: Starting force unmount of GPFS file systems Mon Dec 17 20:56:47 GMT 2012: mmshutdown: Shutting down GPFS daemons Shutting down! Unloading modules from /lib/modules/2.6.32-279.14.1.el6.x86_64/extra Unloading module mmfs26 Unloading module mmfslinux Unloading module tracedev Mon Dec 17 20:56:54 GMT 2012: mmshutdown: Finished [root@snoopy ~]# mmstartup Mon Dec 17 20:56:57 GMT 2012: mmstartup: Starting GPFS ... [root@snoopy ~]# mmlsconfig | grep -i heat fileHeatPeriodMinutes 1 [root@snoopy ~]# echo 
    "abcdef" > /mnt/gpfs/online/test/testfile [root@snoopy online]$ echo 
    "123456" > /mnt/gpfs/online/test/testfile2 [root@snoopy ~]# cat /mnt/gpfs/online/test/testfile abcdef [root@snoopy online]$ /usr/lpp/mmfs/bin/mmapplypolicy /mnt/gpfs/online/test -P heat.pol -L 4 [I] GPFS Current Data Pool Utilization in KB and % fast    2304    2683904 0.085845% nearline 2304    2683904 0.085845% offline 2304    2683904 0.085845% online  32512   2683904 1.211370% system  0       0       0.000000% (no user data) [I] 364053 of 1756672 inodes used: 20.724017%. [I] Loaded policy rules from heat.pol. Evaluating MIGRATE/DELETE/EXCLUDE rules with CURRENT_TIMESTAMP = 2012-12-17@21:24:37 UTC parsed 0 Placement Rules, 0 Restore Rules, 0 Migrate/Delete/Exclude Rules, 1 List Rules, 1 External Pool/List Rules RULE EXTERNAL LIST 
    'genheat' EXEC 
    '/mnt/gpfs/online/listproc'   RULE 
    'heat1' LIST 
    'genheat' WEIGHT(INODE) SHOW(computeFileHeat(CURRENT_TIMESTAMP-ACCESS_TIME,xattr(
    'gpfs.FileHeat'),KB_ALLOCATED)) [I]2012-12-17@21:24:37.316 Directory entries scanned: 3. [I] Directories scan: 2 files, 1 directories, 0 other objects, 0 
    'skipped' files and/or errors. [I]2012-12-17@21:24:37.323 Sorting 3 file list records. /mnt/gpfs/online/test/testfileic RULE 
    'heat1' LIST 
    'genheat' WEIGHT(61442.000000) /mnt/gpfs/online/test/testfile2  RULE 
    'heat1' LIST 
    'genheat' WEIGHT(61453.000000) [I] Inodes scan: 2 files, 1 directories, 0 other objects, 0 
    'skipped' files and/or errors. [I]2012-12-17@21:24:37.328 Policy evaluation. 3 files scanned. [I]2012-12-17@21:24:37.334 Sorting 2 candidate file list records. [I]2012-12-17@21:24:37.335 Choosing candidate files. 2 records scanned. [I] Summary of Rule Applicability and File Choices: Rule#  Hit_Cnt KB_Hit  Chosen  KB_Chosen       KB_Ill  Rule 0     2       0       2       0       0       RULE 
    'heat1' LIST 
    'genheat' WEIGHT(.) SHOW(.)   [I] Filesystem objects with no applicable rules: 1.   [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 0KB: 0 of 0 candidates; Chose to premigrate 0KB: 0 candidates; Already co-managed 0KB: 0 candidates; Chose to delete 0KB: 0 of 0 candidates; Chose to list 0KB: 2 of 2 candidates; Chose to change encryption 0KB: 0 of 0 candidates; 0KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: fast    2304    2683904 0.085845% nearline        2304    2683904 0.085845% offline 2304    2683904 0.085845% online  32512   2683904 1.211370% system  0       0       0.000000% (no user data) LIST /tmp/mmPolicy.ix.32463.4FA9F331.1ution. 0 files dispatched.  \....... [I]2012-12-17@21:24:37.356 Policy execution. 2 files dispatched. [I] A total of 2 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 
    'skipped' files and/or errors. [tucks@snoopy online]$ cat heatmap 61453 456307718 0   -- /mnt/gpfs/online/test/testfile2 61442 2083412008 0   -- /mnt/gpfs/online/test/testfile
    
  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: SHOW(FILE_HEAT) does not elicit a value?

    ‏2012-12-18T00:12:41Z  
    • Tucks
    • ‏2012-12-17T21:33:22Z
    So.. here's a reproduction of the issue.
    I created two test files and cat the contents of one.

    As you can see, no heat value is inserted into the LIST (listproc just copies the list file to a known filename).
    I.E. the values are still NULL (thanks for the NULL pointer (bad pun..) btw. I'll be using that lots).

    I'm expecting both files to have a heat value.
    The file that was read being higher than the other.

    Note: after the mmchconfig and recycling of mmfsd, fileheatlosspercent is not present in mmlsconfig. Weird?

    <pre class="jive-pre"> [root@snoopy ~]# mmchconfig fileheatperiodminutes=1,fileheatlosspercent=10 mmchconfig: Command successfully completed [root@snoopy ~]# mmshutdown Mon Dec 17 20:56:42 GMT 2012: mmshutdown: Starting force unmount of GPFS file systems Mon Dec 17 20:56:47 GMT 2012: mmshutdown: Shutting down GPFS daemons Shutting down! Unloading modules from /lib/modules/2.6.32-279.14.1.el6.x86_64/extra Unloading module mmfs26 Unloading module mmfslinux Unloading module tracedev Mon Dec 17 20:56:54 GMT 2012: mmshutdown: Finished [root@snoopy ~]# mmstartup Mon Dec 17 20:56:57 GMT 2012: mmstartup: Starting GPFS ... [root@snoopy ~]# mmlsconfig | grep -i heat fileHeatPeriodMinutes 1 [root@snoopy ~]# echo "abcdef" > /mnt/gpfs/online/test/testfile [root@snoopy online]$ echo "123456" > /mnt/gpfs/online/test/testfile2 [root@snoopy ~]# cat /mnt/gpfs/online/test/testfile abcdef [root@snoopy online]$ /usr/lpp/mmfs/bin/mmapplypolicy /mnt/gpfs/online/test -P heat.pol -L 4 [I] GPFS Current Data Pool Utilization in KB and % fast 2304 2683904 0.085845% nearline 2304 2683904 0.085845% offline 2304 2683904 0.085845% online 32512 2683904 1.211370% system 0 0 0.000000% (no user data) [I] 364053 of 1756672 inodes used: 20.724017%. [I] Loaded policy rules from heat.pol. Evaluating MIGRATE/DELETE/EXCLUDE rules with CURRENT_TIMESTAMP = 2012-12-17@21:24:37 UTC parsed 0 Placement Rules, 0 Restore Rules, 0 Migrate/Delete/Exclude Rules, 1 List Rules, 1 External Pool/List Rules RULE EXTERNAL LIST 'genheat' EXEC '/mnt/gpfs/online/listproc' RULE 'heat1' LIST 'genheat' WEIGHT(INODE) SHOW(computeFileHeat(CURRENT_TIMESTAMP-ACCESS_TIME,xattr( 'gpfs.FileHeat'),KB_ALLOCATED)) [I]2012-12-17@21:24:37.316 Directory entries scanned: 3. [I] Directories scan: 2 files, 1 directories, 0 other objects, 0 'skipped' files and/or errors. [I]2012-12-17@21:24:37.323 Sorting 3 file list records. /mnt/gpfs/online/test/testfileic RULE 'heat1' LIST 'genheat' WEIGHT(61442.000000) /mnt/gpfs/online/test/testfile2 RULE 'heat1' LIST 'genheat' WEIGHT(61453.000000) [I] Inodes scan: 2 files, 1 directories, 0 other objects, 0 'skipped' files and/or errors. [I]2012-12-17@21:24:37.328 Policy evaluation. 3 files scanned. [I]2012-12-17@21:24:37.334 Sorting 2 candidate file list records. [I]2012-12-17@21:24:37.335 Choosing candidate files. 2 records scanned. [I] Summary of Rule Applicability and File Choices: Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule 0 2 0 2 0 0 RULE 'heat1' LIST 'genheat' WEIGHT(.) SHOW(.) [I] Filesystem objects with no applicable rules: 1. [I] GPFS Policy Decisions and File Choice Totals: Chose to migrate 0KB: 0 of 0 candidates; Chose to premigrate 0KB: 0 candidates; Already co-managed 0KB: 0 candidates; Chose to delete 0KB: 0 of 0 candidates; Chose to list 0KB: 2 of 2 candidates; Chose to change encryption 0KB: 0 of 0 candidates; 0KB of chosen data is illplaced or illreplicated; Predicted Data Pool Utilization in KB and %: fast 2304 2683904 0.085845% nearline 2304 2683904 0.085845% offline 2304 2683904 0.085845% online 32512 2683904 1.211370% system 0 0 0.000000% (no user data) LIST /tmp/mmPolicy.ix.32463.4FA9F331.1ution. 0 files dispatched. \....... [I]2012-12-17@21:24:37.356 Policy execution. 2 files dispatched. [I] A total of 2 files have been migrated, deleted or processed by an EXTERNAL EXEC/script; 0 'skipped' files and/or errors. [tucks@snoopy online]$ cat heatmap 61453 456307718 0 -- /mnt/gpfs/online/test/testfile2 61442 2083412008 0 -- /mnt/gpfs/online/test/testfile </pre>
    It should also be mentioned, that the policy engine only reads stuff that has been synced to disk. So doing things that still leave dirty info in memory will not show up immediately in a policy scan output.

    You can force everything everywhere to disk by using mmfsctl suspend/resume and then run the policy query.
  • Tucks
    Tucks
    78 Posts

    Re: SHOW(FILE_HEAT) does not elicit a value?

    ‏2012-12-18T06:31:31Z  
    • dlmcnabb
    • ‏2012-12-18T00:12:41Z
    It should also be mentioned, that the policy engine only reads stuff that has been synced to disk. So doing things that still leave dirty info in memory will not show up immediately in a policy scan output.

    You can force everything everywhere to disk by using mmfsctl suspend/resume and then run the policy query.
    suspend/resume makes no difference.

    I.E
    write files
    suspend & resume
    read one file many times
    do not read another file
    regenerate policy LIST

    The value is not being generated.
    Am I doing something wrong, or does this not actually work atm.? (hey, it's new..)
  • Tucks
    Tucks
    78 Posts

    Re: SHOW(FILE_HEAT) does not elicit a value?

    ‏2012-12-18T06:36:31Z  
    • Tucks
    • ‏2012-12-18T06:31:31Z
    suspend/resume makes no difference.

    I.E
    write files
    suspend & resume
    read one file many times
    do not read another file
    regenerate policy LIST

    The value is not being generated.
    Am I doing something wrong, or does this not actually work atm.? (hey, it's new..)
    To be more specific, using Marc's NULL method:

    
    [tucks@snoopy online]$ cat heatmap 61453 456307718 0 ++ _NULL_ -- /mnt/gpfs/online/test/testfile2 61442 2083412008 0 ++ _NULL_ -- /mnt/gpfs/online/test/testfile
    


    This is after making sure both files are syncd to disk.
    testfile has been read 100 times.
    testfile2 has never been read.
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: SHOW(FILE_HEAT) does really show file heat!!

    ‏2012-12-18T16:38:26Z  
    • Tucks
    • ‏2012-12-18T06:36:31Z
    To be more specific, using Marc's NULL method:

    <pre class="jive-pre"> [tucks@snoopy online]$ cat heatmap 61453 456307718 0 ++ _NULL_ -- /mnt/gpfs/online/test/testfile2 61442 2083412008 0 ++ _NULL_ -- /mnt/gpfs/online/test/testfile </pre>

    This is after making sure both files are syncd to disk.
    testfile has been read 100 times.
    testfile2 has never been read.
    FILE_HEAT is a measure of I/O activity against the data blocks of your files.
    Tiny files are now stored completely in the inode, so their heat is always NULL!

    Here is an example from one of my test systems. While experimenting to create this example, I played with file giggish more, so it is very hot.

    {code}

    root@fin44 heat-test]# dd if=/dev/urandom bs=1024 count=1024 of=meggish
    1024+0 records in
    1024+0 records out
    1048576 bytes (1.0 MB) copied, 0.202841 s, 5.2 MB/s
    root@fin44 heat-test]# ls -l *
    -rw-r--r-- 1 root root 1073741824 Dec 18 08:25 giggish
    -rw-r--r-- 1 root root 1048576 Dec 18 08:30 meggish
    -rw-r--r-- 1 root root 5 Dec 18 08:27 tiny
    -rw-r--r-- 1 root root 0 Dec 18 08:27 zippo

    root@fin44 heat-test]# cat * * * * * * >/dev/null
    root@fin44 heat-test]# mmfsctl abc suspend; mmfsctl abc resume
    Writing dirty data to disk
    Quiescing all file system operations
    Writing dirty data to disk again
    Resuming operations.

    root@fin44 heat-test]# mmapplypolicy /abc/heat-test -P /ghome/makaplan/policies/fileheat-simp.policy -I defer -f /tmp/fh ; cat /tmp/fh.list.fh

    ...

    59027465 450632253 0 ++ +1.49929199218750E+001 -- /abc/heat-test/giggish
    59027460 622421516 0 ++ +2.50000000000000E+000 -- /abc/heat-test/meggish
    59027458 1958692162 0 ++ NULL -- /abc/heat-test/zippo
    59027459 2050869430 0 ++ NULL -- /abc/heat-test/tiny
  • Tucks
    Tucks
    78 Posts

    Re: SHOW(FILE_HEAT) does not elicit a value?

    ‏2012-12-21T18:37:08Z  
    Hey. I now have many hot files. Niiiiiiiice.
    Shame it's > inode size, but I get that.
    Would have been marvelous to do <= inode size for render farms 'acceleration'.
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: SHOW(FILE_HEAT) does not elicit a value?

    ‏2012-12-21T18:57:24Z  
    • Tucks
    • ‏2012-12-21T18:37:08Z
    Hey. I now have many hot files. Niiiiiiiice.
    Shame it's > inode size, but I get that.
    Would have been marvelous to do <= inode size for render farms 'acceleration'.
    The main objective of tracking file temperature is to provide actionable information to sysadmin. Satisfying one's curiosity is important too, but arguably not as much. Suppose you know for a fact that a tiny data-in-inode file is hot. This is good to know, but what could you actually do about it? File data is stored inside an inode header, i.e. it resides in the system pool, on a metadata disk. You can't migrate it to another pool, and you already know that it's good for performance to keep metadata on fast disk, and there's no practical way (or need) to control the disk placement of individual inodes. On the other hand, maintaining file temperature involves a certain amount of overhead. So the benefit is not worth the cost.

    yuri