I have 2 types of streaming input data as below:
Start message schema: MSISDN, IMEI
Stop message schema: MSISDN, IMSI
I want to process both streams and extract output as MSISDN, IMEI, IMSI
I have found this can be done using join operator as output port is mutating. Also using a custom operator where we can use a Map.
In case of join we are planning to use below window configuration:
Start message : port 1 : sliding, time (1800)
Start message : port 2 : sliding, count (1)
Using above we shall keep Start message for some time and once we get stop messages join operation will be performed and output will be sent.
In case of custom operator we shall use 2 input ports
Start message port 1
Stop message port 2
We shall insert all tuples from port 1 into a map. Once we have a tuple in port 2 we shall check in map and send a output along with removal of matching key removal in map.
From both functionality join or custom using map, which one will have good performance.
Or is there any other good way to process this ?