Pinned topic Q) increasing log file processing in real time by Streams
The source data is application log file, this has some Apps's activity which is created by mobile user in real time.
user --> Apps on smart phone --> activity collecting server (writing all user's activity in real time) --> a log file (very fast increased) --> Streams reading that log file in real time ==> statistics & analytics
My question is....
Can Streams read and process the log file when server application is writing a lot data to that log file ?
I heard, writing and reading are exclusive mode by OS like Linux, so it's impossible.
If then,,,, what is our solution or any tips to process this logfile in real time by Streams ? Adapter or API or small interface program by user or change some data flow architecture .....?
SystemAdmin 110000D4XK1245 Posts
Re: Q) increasing log file processing in real time by Streams2013-02-10T15:59:55ZThis is the accepted answer. This is the accepted answer.Would a FileSource() with hotFile=true work in that scenario?
Armand_ruiz 270004S4PY2 Posts
Kevin_Foster 120000GMHC98 Posts
Re: Q) increasing log file processing in real time by Streams2013-08-15T18:47:27ZThis is the accepted answer. This is the accepted answer.
- MikeSpicer 060000C4DT
Another possibly more direct method would be to tap the Log4j software in most (?) web and application servers with a wire-based feed into a TCPSource or UDPSource operator:
it seems to be possible to use JMS also, but I would think that this store-and-forward overhead would be unnecessary within a single computer room. Maybe over long distances though....