Topic
4 replies Latest Post - ‏2013-08-15T18:47:27Z by Kevin_Foster
SystemAdmin
SystemAdmin
1245 Posts
ACCEPTED ANSWER

Pinned topic Q) increasing log file processing in real time by Streams

‏2013-02-08T03:54:34Z |
I am doing a PoC, one of secenario is real time collecting and analytics.
The source data is application log file, this has some Apps's activity which is created by mobile user in real time.

user --> Apps on smart phone --> activity collecting server (writing all user's activity in real time) --> a log file (very fast increased) --> Streams reading that log file in real time ==> statistics & analytics

My question is....
Can Streams read and process the log file when server application is writing a lot data to that log file ?

I heard, writing and reading are exclusive mode by OS like Linux, so it's impossible.
If then,,,, what is our solution or any tips to process this logfile in real time by Streams ? Adapter or API or small interface program by user or change some data flow architecture .....?

Thanks..
  • SystemAdmin
    SystemAdmin
    1245 Posts
    ACCEPTED ANSWER

    Re: Q) increasing log file processing in real time by Streams

    ‏2013-02-10T15:59:55Z  in response to SystemAdmin
    Would a FileSource() with hotFile=true work in that scenario?
    • Armand_ruiz
      Armand_ruiz
      2 Posts
      ACCEPTED ANSWER

      Re: Q) increasing log file processing in real time by Streams

      ‏2013-07-23T15:39:13Z  in response to SystemAdmin

      I am also interested on the answer of this question...anyone can say something about it?

      Thanks!

    • MikeSpicer
      MikeSpicer
      18 Posts
      ACCEPTED ANSWER

      Re: Q) increasing log file processing in real time by Streams

      ‏2013-08-13T16:56:33Z  in response to SystemAdmin

      I believe that hotfile=true should be able to handle this.

      • Kevin_Foster
        Kevin_Foster
        98 Posts
        ACCEPTED ANSWER

        Re: Q) increasing log file processing in real time by Streams

        ‏2013-08-15T18:47:27Z  in response to MikeSpicer

        Another possibly more direct method would be to tap the Log4j software in most (?) web and application servers with a wire-based feed into a TCPSource or UDPSource operator:

        http://logging.apache.org/log4j/2.x/manual/appenders.html

        it seems to be possible to  use JMS also, but I would think that this store-and-forward overhead would be unnecessary within a single computer room. Maybe over long distances though....

        -Kevin