Move Container can fail with ANR3682W and ANR1893E

Troubleshooting

Problem

Move Container can fail with ANR3682W indicating that the container is empty when the command ‘QUERY CONTAINER F=D’ output shows that the container still contains a small amount of data.

Symptom

MOVE CONTAINER process can fail with the following errors:

ANR3682W Container /full/path/to/0000000000006b17.dcf is empty and cannot be moved.
ANR0985I Process xxx for Move Container running in the BACKGROUND completed with completion state FAILURE at <timestamp>.
ANR1893E Process xxx for Move Container completed with a completion state of FAILURE.

The command ‘QUERY CONTAINER /full/path/to/0000000000006b17.dcf F=D’ output shows that the container still contains data with only 1MB size, for example:
Maximum size (MB): 10,240
Free Space (MB): 10,239

The container has been at the above size for a long time which looks like it should have been empty but never been reused since long ago.

Cause

The container is still in the global in-use list determined by the background container deletion thread for IBM Spectrum Protect server.

Environment

IBM Spectrum Protect Server on all platforms

Diagnosing The Problem

In the Spectrum Protect server trace with flag SDCNTR SQL, it will show the messages as below:

16:19:44.470 [68][tbcol.c][2800][TbPrintRowToTrace]:Row addr=(1253ef858), 6 columns:
16:19:44.470 [68][tbcol.c][2694][TbPrintColToTrace]: no name( 0):INT32 27415
16:19:44.471 [68][tbcol.c][2694][TbPrintColToTrace]: no name( 1):STRING 0x1253efc78(/full/path/to/0000000000006b17.dcf) len=45
16:19:44.471 [68][tbcol.c][2694][TbPrintColToTrace]: no name( 2):INT8 1
16:19:44.471 [68][tbcol.c][2694][TbPrintColToTrace]: no name( 3):UINT32 5
16:19:44.471 [68][tbcol.c][2694][TbPrintColToTrace]: no name( 4):NULL COL
16:19:44.471 [68][tbcol.c][2694][TbPrintColToTrace]: no name( 5):DATETIMEN 2017-09-10 19:13:24.168914
16:19:44.471 [68][tbrsql.c][2057][tbRegSkipAndFetch]:exiting with rc=TBRC_OK
16:19:44.472 [68][sdcntr.c][6104][SdFindGlobalListContainer]:Exit: rc=0, cntrP 11a679510 (27415).
16:19:44.473 [68][sdcntr.c][12740][DeleteContainers]:Container id 27415 is in-use. Cannot delete.
16:19:44.473 [68][sdcntr.c][13152][DeleteContainers]:Container 0000000000006b17.dcf is still being referenced (rc=4401). Skipping

The above indicates that the container deletion thread is working properly as it has already detected the container 0000000000006b17.dcf. However the container 0000000000006b17.dcf is still in the global inuse list. So the container cannot be deleted at that point.

Resolving The Problem

MOVE CONTAINER is mainly used to move contents of a container to another container if a storage pool directory is removed or if a container is damaged. Deleting empty containers is always handled by background container deletion thread itself for the Spectrum Protect server and usually needs some time because the background thread wakes up in interval to verify deletion qualification for a container before removing it. So it is expected to see time delay for removing empty container.

Document Location

Worldwide

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSEQVQ","label":"IBM Spectrum Protect"},"Component":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]

Product Synonym

TSM Tivoli Storage Manager

Was this topic helpful?

Document Information

Modified date:
02 November 2020

UID

ibm11136326

Tips