In the InfoCenter documentation, the HDFSFileSink operator in the bigdata toolkit (I am using version 1.0.2 of the bigdata toolkit) refers to a parameter "ignoreWindowMarkers". This parameter doesn't show up in the 'param' tab of my 'edit' box for the operator, and if I try to manually add the param in the SPL I get a compile error that there isn't an "ignoreWindowMarkers" param.
Does anybody know if this was supported in a previouis version, or if it's expected for a future version?
Since I can't use the ignoreWindowMarkers in my code, I have an extra step where I am using a Custom operator to snip out Window punctuations:
(stream<FinTransResult> ProcessedStreamWithoutWindowPunctuations) as
RemoveWindowPunctuations = Custom(FinancialTransactionResults)
onTuple FinancialTransactionResults : submit(FinancialTransactionResults,
onPunct FinancialTransactionResults : if(currentPunct() !=
and if possible, I'd rather not have custom code to do this (but without it, the HDFSFileSink is writing (overwriting, since it can't append)) on each individual transaction because I happen to have a window size of 1 transaction because of previous join logic, etc.
Any suggestions or information on how to get "ignoreWindowMarkers" working?
thanks in advance,