I've discovered a performance issue with our HSM workflow.
When we hit an individual stgpool THRESHOLD callback we generate a list of candidates to migrate to TSM LTO5 via an oldest first policy.
RULE 'yadayada' MIGRATE FROM POOL 'mypool' THRESHOLD (95,85,70) WEIGHT((DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) TO POOL 'hsmpool' WHERE some matching rules..
The above creates a candidate list which is sorted according to WEIGHT((DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)))
Candidates are supplied to TSM in -B sized batches in the order of the sort.
To recall we generate a file list via:
find /path/to/dir -type f -print > filelist dsmrecall -Detail -FI=filelist
However on recall tape shoe-shining occurs as the filelist is not representative of the order that files were migrated to tape. I.E. the sort order generated via GPFS.
1. Can we post-process the sort of GPFS candidates to perform a further sort -n (we have sequences of images) before the batches are passed to dsmmigrate?
2. Alternatively, we could sort our find /path/to/dir results to match the GPFS sort. How do we guarantee a post-sort on the find would match the sorted order of the GPFS supplied candidate list?
3. I have a gut feeling (I.E. still looking into this) that this may be compounded by the fact that dsmmigrate is supplied in -B batchsize and different nodes could supply these out of order? Perhaps defer is the way to go?
The crux of it is that in order to have efficient streaming off tape with no/minimal shoe-shining, the files must be dsmrecalled in the order they were written to tape.