Troubleshooting
Problem
On a 2 node configuration, the job partitions the data using Round Robin and sorts on the unique key of the data, but sometimes the last record is placed on the same node as the second to last record.
Symptom
The job consists of the following stages: <Data Set> --> <Transformer> --> <Filter> --> <Data Set>
It passes the starting number and count of records in the initial data set into the job as a variables. The transformer partitions the data using Round Robin and sorts on the unique key of the data. It uses a transform variable to calculate the Next Number for each record:
((@PARTITIONNUM+(@NUMPARTITIONS*(@INROWNUM-1))+1)+NextBatchNmbr)
The transform also calculates a field with the maximum value allowed for the Next Number by adding the initial Next Number and Record Count. The filter is used to reject any records where the Next Number is larger than it is expected to be.
An instance has been observed where the last record is placed on the same node as the second to last record causing the calculated number to skip a value and results in the last record being rejected.
Log InLog in to view more of this document
Was this topic helpful?
Document Information
Modified date:
25 April 2025
UID
swg21612938