Asset/Flow Upload to IGC Fails with 500: Internal Error
Problem
The last step of Manta Flow for IGC (upload of the lineage to IGC) occasionally fails with the following error in any of the
*IgcUploadMetadataMasterScenario_*properties.log
log files.
2017-04-10 16:09:26.504 [pool-2-thread-1] INFO eu.profinit.manta.dataflow.igc.common.UploadModelTask - Inserting flows.
2017-04-10 16:12:35.358 [pool-2-thread-1] ERROR eu.profinit.manta.connector.http.caller.callback.BasicErrorCallback - Response 500: {"message":"Internal Error","code":500}.
Additional Information
Based on the information about HTTP status codes available in
InfoSphere Information Governance Catalog REST API: Tips,
tricks, and time-savers (the attached PDF), additional detailed information about the error is available in the Websphere log files
SystemOut.log
, SystemError.log
, and igc.log
. The location of these log files in the filesystem is described in detail in IBM Technote
InfoSphere Information Governance Catalog: log files.
A typical case is low limits for objects to be processed by IGC through API. The corresponding error message is:
2017-04-10T16:12:36,084 [WebContainer : 3] INFO com.ibm.is.igc.rest.v1.providers.APIExceptionMapper - com.ibm.is.vr.client.query.VRReachedInMemoryProcessingLimitException:
Reached limit of processed in-memory objects.
The current request has caused 27201 objects to be loaded into memory.
Your options:
a. Ask your WebSphere/InfoSphere system administrator to raise the JVM heap size and maxObjectsInMemory.
b. Run your task using the Command Line Inteface (CLI).
c. Retry your request at a later point in time, when there are less concurrent requests.
(This message is for logging/debug purposes only. The UI should compose its own, translatable text.)
Solution
Investigate the error message in the above mentioned log files. If the message says "Reached limit of processed in-memory objects", the solution is:
- Reduce the batch size for upload to IGC as described in the article Common Resource Configuration for IGC in the section called "IBM IGC Connection Properties."
- For previous versions of Manta, increase the limits as described in, for example, Prepare Information Governance Catalog for Use with IBM Industry Models.
Otherwise, check for solutions in the IBM Knowledge Center or with IBM Support.