Pinned topic Q) increasing log file processing in real time by Streams
The source data is application log file, this has some Apps's activity which is created by mobile user in real time.
user --> Apps on smart phone --> activity collecting server (writing all user's activity in real time) --> a log file (very fast increased) --> Streams reading that log file in real time ==> statistics & analytics
My question is....
Can Streams read and process the log file when server application is writing a lot data to that log file ?
I heard, writing and reading are exclusive mode by OS like Linux, so it's impossible.
If then,,,, what is our solution or any tips to process this logfile in real time by Streams ? Adapter or API or small interface program by user or change some data flow architecture .....?
SystemAdmin 110000D4XK1245 PostsACCEPTED ANSWER
Re: Q) increasing log file processing in real time by Streams2013-02-10T15:59:55Z in response to SystemAdminWould a FileSource() with hotFile=true work in that scenario?
MikeSpicer 060000C4DT18 PostsACCEPTED ANSWER
Re: Q) increasing log file processing in real time by Streams2013-08-13T16:56:33Z in response to SystemAdmin
I believe that hotfile=true should be able to handle this.
Kevin_Foster 120000GMHC98 PostsACCEPTED ANSWER
Re: Q) increasing log file processing in real time by Streams2013-08-15T18:47:27Z in response to MikeSpicer
Another possibly more direct method would be to tap the Log4j software in most (?) web and application servers with a wire-based feed into a TCPSource or UDPSource operator:
it seems to be possible to use JMS also, but I would think that this store-and-forward overhead would be unnecessary within a single computer room. Maybe over long distances though....