Topic
  • 2 replies
  • Latest Post - ‏2013-01-23T17:52:33Z by SystemAdmin
SystemAdmin
SystemAdmin
2092 Posts

Pinned topic policy rules - list and delete - how to exclude the same file

‏2013-01-23T16:30:19Z |
Hello,

I use gpfs policies and rules to delete old files. I use several "exclude" rules to exclude certain files from deletion. Before I actually do delete I want to create a list of all files that will be deleted some days in advance to inform the users.

What I don't know: How can I apply the same "exclude" rules when I create the list? I really want to use the same rules for exclusion to make 100% sure that what I send to the users is really what I'll delete later. I don't want to keep manually two different syntaxes in sync. I would welcome and hint or example.

Example - the deletion rules look like:

Rule "XcludeTXT" EXCLUDE WHERE SUBSTR(NAME,LENGTH(NAME)-3)='.txt'
Rule "XcludeSH" EXCLUDE WHERE SUBSTR(NAME,LENGTH(NAME)-2)='.sh'

RULE "OldFiles"
DELETE
WHERE PATH_NAME LIKE '%/Data10/e_____/%'
AND DAYS(CURRENT_TIMESTAMP) - DAYS(MODIFICATION_TIME) > 60

How do I create a file list which lists the files to be deleted using the same Rules for exclusion? Currently I create a list of all files and do the exclusion by name and age later with some postprocessing. I create a list with some additional data using SHOW().

My current list rule file looks like:

define(toSeconds,(($1) SECONDS(12,6)))
define(toUnixSeconds,)

RULE

LIST 'list'
SHOW(varchar(USER_ID) || ' ' || varchar( toUnixSeconds(modification_time) ) || ' '
|| varchar (file_size)
)

RULE
EXTERNAL LIST 'list'
EXEC '/scratch/script.sh'
The script 'script.sh' just dumps the file lists to individual files.

Kind regards,

Heiner Billich
Updated on 2013-01-23T17:52:33Z at 2013-01-23T17:52:33Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    2092 Posts

    correction

    ‏2013-01-23T16:43:06Z  
    Hello,

    the m4 macros in the list rules don't show up correctly in my original post. I'm sorry for that, I don't know how to fix it. But it make no difference regarding my question ...

    Sorry,

    Heiner

    I try once more:
     define(
    toSeconds,
    (($1) SECONDS(12,6)))
    define( toUnixSeconds,)

    RULE

    LIST 'list'
    SHOW(varchar(USER_ID) || ' ' || varchar( toUnixSeconds(modification_time) ) || ' '
    || varchar (file_size)
    )

    RULE
    EXTERNAL LIST 'list'
    EXEC '/scratch/2013-01-17-X02DA-file-list/script.sh'
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: correction

    ‏2013-01-23T17:52:33Z  
    Hello,

    the m4 macros in the list rules don't show up correctly in my original post. I'm sorry for that, I don't know how to fix it. But it make no difference regarding my question ...

    Sorry,

    Heiner

    I try once more:
     define(
    toSeconds,
    (($1) SECONDS(12,6)))
    define( toUnixSeconds,)

    RULE

    LIST 'list'
    SHOW(varchar(USER_ID) || ' ' || varchar( toUnixSeconds(modification_time) ) || ' '
    || varchar (file_size)
    )

    RULE
    EXTERNAL LIST 'list'
    EXEC '/scratch/2013-01-17-X02DA-file-list/script.sh'
    Use LIST rules with the mmapplypolicy -I defer option to create a list of files that are candidates for deletion.
    You can edit the filelist as you please. When you and your users are satisfied with the list, run a script against it to do the deletions.

    For example, your policy rule file would look like this:
    
    rule 
    'a' external list 
    'x' exec 
    '/ghome/makaplan/policies/doer.sh' opts 
    'CMD'   rule 
    'bx' list 
    'x' EXCLUDE WHERE  criteria-to-avoid-deletion-even-if-file-would-match-the-next-rule rule 
    'bi' list 
    'x'  WHERE criteria-for-deletion
    


    You would issue mmapplypolicy like this:
    
    mmapplypolicy /directory-to-explore  -P /path-to-policy-rules -I defer -f /tmp/xcaution
    


    The doer.sh script could look like this:
    
    
    
    if [ $1 = TEST ] then echo TEST OKAY. exit 0 fi   export filelist=$2 export cmd=$3   # execute cmd against input list x=
    "$*" 
    
    while read inode gen some opt path otherstuff ; 
    
    do #   echo 
    "$cmd $path"  # remove comment to see what is going on $cmd $path done  < $filelist
    


    You could view the filelist like this:
    
    ./doer.sh list /tmp/xcaution.list.x  
    "ls -ld"
    


    After you are sure that is the list of files you wish to delete, throw caution to the wind and do this!
    
    ./doer.sh list /tmp/xcaution.list.x  
    "rm -f"