Topic
  • 11 replies
  • Latest Post - ‏2014-06-20T10:34:54Z by networkingkool
MAKOLI
MAKOLI
19 Posts

Pinned topic Apache Access Logs

‏2013-07-08T07:03:04Z |

I was trying to collect Apache access and error logs. My configuration was:



# Apache access file:


$ModLoad imfile

$InputFileName /var/log/apache2/access.log

$InputFileTag apache-access:

$InputFileStateFile stat-apache-access

$InputFileSeverity info

$InputRunFileMonitor


#Apache Error file:


$InputFileName /var/log/apache2/error.log

$InputFileTag apache-errors:

$InputFileStateFile stat-apache-error

$InputFileSeverity error

$InputRunFileMonitor


$InputFilePollInterval 10


if $programname == 'apache-access' then @192.168.x.x

if $programname == 'apache-errors' then @192.168.x.x
 

Now it shows the Event Name as "Linux login messages Message" and Low Level Category "stored".

Why is it so?

Also is there any way to collect the access logs in real time? I did it with error logs but cant be able to do so with access logs.

  • Aaron_Breen(IBM)
    Aaron_Breen(IBM)
    124 Posts

    Re: Apache Access Logs

    ‏2013-07-09T05:08:59Z  

    What version of Apache? can you provide a scrubbed anonymous example of 1 event? 

    This is the second time I have heard this in a week. I am wondering if both are configurations or if we have a new apache event format to parse. 

  • MAKOLI
    MAKOLI
    19 Posts

    Re: Apache Access Logs

    ‏2013-07-09T06:41:39Z  

    What version of Apache? can you provide a scrubbed anonymous example of 1 event? 

    This is the second time I have heard this in a week. I am wondering if both are configurations or if we have a new apache event format to parse. 

    Apache's Version:

    Server version: Apache/2.2.14 (Ubuntu)
    Server built:   Mar  5 2012 16:42:17

    Sample error log:

    <131>Jul  9 11:17:59 xxxxx apache-errors: [Tue Jul 09 11:07:00 2013] [notice] Apache/2.2.14 (Ubuntu) PHP/5.3.2-1ubuntu4.17 with Suhosin-Patch configured -- resuming normal operations

    Sample access log:

    <134>Jul  9 11:33:00 xxxx apache-access: 192.168.6.121 - - [09/Jul/2013:11:32:56 +0500] "GET / HTTP/1.1" 200 622 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36"

  • MAKOLI
    MAKOLI
    19 Posts

    Re: Apache Access Logs

    ‏2013-07-09T06:45:31Z  

    Another thing I noticed just now.

    Apaches access logs can be successfully parsed if the log source type is "Websense V Series". But cant able to find a way to parse apace-errors logs as they are still showing as Login messages.

  • Aaron_Breen(IBM)
    Aaron_Breen(IBM)
    124 Posts

    Re: Apache Access Logs

    ‏2013-07-10T14:13:59Z  
    • MAKOLI
    • ‏2013-07-09T06:45:31Z

    Another thing I noticed just now.

    Apaches access logs can be successfully parsed if the log source type is "Websense V Series". But cant able to find a way to parse apace-errors logs as they are still showing as Login messages.

    I have one other customer with a very similar issue. This should be logged as a support ticket (PMR) as there is going to need to be a code update to the DSM

  • Jeff Rusk (IBM)
    Jeff Rusk (IBM)
    5 Posts

    Re: Apache Access Logs

    ‏2013-07-16T17:51:35Z  
    • MAKOLI
    • ‏2013-07-09T06:41:39Z

    Apache's Version:

    Server version: Apache/2.2.14 (Ubuntu)
    Server built:   Mar  5 2012 16:42:17

    Sample error log:

    <131>Jul  9 11:17:59 xxxxx apache-errors: [Tue Jul 09 11:07:00 2013] [notice] Apache/2.2.14 (Ubuntu) PHP/5.3.2-1ubuntu4.17 with Suhosin-Patch configured -- resuming normal operations

    Sample access log:

    <134>Jul  9 11:33:00 xxxx apache-access: 192.168.6.121 - - [09/Jul/2013:11:32:56 +0500] "GET / HTTP/1.1" 200 622 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36"

    Can you confirm whether the custom log format was configured as per docs?

    Step 3 Add the following information in the Apache configuration file to specify the custom
    log format:
    LogFormat "%h %A %l %u %t \"%r\" %>s %p %b" <log format name>

    Not following this format could impact parsing.

  • MAKOLI
    MAKOLI
    19 Posts

    Re: Apache Access Logs

    ‏2013-07-17T07:34:49Z  

    Can you confirm whether the custom log format was configured as per docs?

    Step 3 Add the following information in the Apache configuration file to specify the custom
    log format:
    LogFormat "%h %A %l %u %t \"%r\" %>s %p %b" <log format name>

    Not following this format could impact parsing.

    I don't think that format given in the documents (the one you mentioned above) is correct for these type of logs.

    As you can clearly see the log format begins with "%h %A" which means "Remote Hostname or IP address and Local IP Address".

    Clearly we can see that in the log entries there is only one IP address i.e. remote.

    Am I missing something or the log format has changed for apache2?

  • networkingkool
    networkingkool
    12 Posts

    Re: Apache Access Logs

    ‏2014-05-14T16:08:10Z  

    I have one other customer with a very similar issue. This should be logged as a support ticket (PMR) as there is going to need to be a code update to the DSM

    Hi,

    I am collecting the log from Apache also. Apache have two log type error and access log. My question is does the guide to collect log from Apache in DSM configuration guide include both types of log?

    One more thing, our Apache web server often hosts many websites (virtualhosts, virtual directories). If I just follow the guide in DSM guide, it won't work. I have to create each virtualhosts, virtual directories one LogFormat and Custom Log settings. I think Qradar document also need to take a note about this.

    Thanks

  • Alaa Ali
    Alaa Ali
    30 Posts

    Re: Apache Access Logs

    ‏2014-06-11T13:42:51Z  

    Hi,

    I am collecting the log from Apache also. Apache have two log type error and access log. My question is does the guide to collect log from Apache in DSM configuration guide include both types of log?

    One more thing, our Apache web server often hosts many websites (virtualhosts, virtual directories). If I just follow the guide in DSM guide, it won't work. I have to create each virtualhosts, virtual directories one LogFormat and Custom Log settings. I think Qradar document also need to take a note about this.

    Thanks

    The DSM guide only shows the configuration for collecting access logs (the CustomLog directive). To send error logs (the ErrorLog directive) through syslog, you can just use the same method. All you'll need to do is change the ErrorLog directive to be this:

    ErrorLog "|/usr/bin/logger -t httpd -p local1.info"

    This will pipe the error logs to the logger facility to log under local1.info. I am not sure if QRadar supports events from the error.log. But if it doesn't, sending the error logs while already sending the access logs will have them both come up under a single log source with the error logs being "Stored" or "Unknown" events. You can fix that by writing an LSX and creating QIDs for the error logs.

    Also, for your virtualhosts, you can place only one of each LogFormat and CustomLog directive, outside your VirtualHost tags. This will make all virtual hosts log to the same file. This is bad though because there will be no way to identify which log is from which site since everything is now in one file (read more about Virtual Hosts and logging in Apache here). We can get around that pretty nicely by changing the log format to include the %v option, but that will mess up the log format that QRadar wants. If the DSM is designed to be very tight, the logs might not be parsed correctly.

  • networkingkool
    networkingkool
    12 Posts

    Re: Apache Access Logs

    ‏2014-06-11T14:17:02Z  
    • Alaa Ali
    • ‏2014-06-11T13:42:51Z

    The DSM guide only shows the configuration for collecting access logs (the CustomLog directive). To send error logs (the ErrorLog directive) through syslog, you can just use the same method. All you'll need to do is change the ErrorLog directive to be this:

    ErrorLog "|/usr/bin/logger -t httpd -p local1.info"

    This will pipe the error logs to the logger facility to log under local1.info. I am not sure if QRadar supports events from the error.log. But if it doesn't, sending the error logs while already sending the access logs will have them both come up under a single log source with the error logs being "Stored" or "Unknown" events. You can fix that by writing an LSX and creating QIDs for the error logs.

    Also, for your virtualhosts, you can place only one of each LogFormat and CustomLog directive, outside your VirtualHost tags. This will make all virtual hosts log to the same file. This is bad though because there will be no way to identify which log is from which site since everything is now in one file (read more about Virtual Hosts and logging in Apache here). We can get around that pretty nicely by changing the log format to include the %v option, but that will mess up the log format that QRadar wants. If the DSM is designed to be very tight, the logs might not be parsed correctly.

    Hi,

    Thanks for your response. I also think like you with Error log issue. But I have no chance to know whether Qradar support Error log format.

    To the second thing, I enable log in each virtualhost directories. It works well. I also add two more fields %v and %{local}p for virtualhost and client port. Actually, I need to write a LSX for pickup the source port.

    I remained one issue with user field %u. Our log never contain the user information with some websites has authentication information. Do you have any experience about this, Ali?

    Thanks

    P/S: I had experienced a bad performance in web access when configure apache log for Qradar. The log not only push directly to Qradar, but also write all to message files. In CustomLog setting, we push all to local1.info log facility, however in rsyslog setting we have a default setting to write down *.info to /var/log/message. I need to add local1.none in that statement to prevent write log to message file, then also prevent the whole web server from performance downgrade.

     

  • Alaa Ali
    Alaa Ali
    30 Posts

    Re: Apache Access Logs

    ‏2014-06-14T07:41:29Z  

    Hi,

    Thanks for your response. I also think like you with Error log issue. But I have no chance to know whether Qradar support Error log format.

    To the second thing, I enable log in each virtualhost directories. It works well. I also add two more fields %v and %{local}p for virtualhost and client port. Actually, I need to write a LSX for pickup the source port.

    I remained one issue with user field %u. Our log never contain the user information with some websites has authentication information. Do you have any experience about this, Ali?

    Thanks

    P/S: I had experienced a bad performance in web access when configure apache log for Qradar. The log not only push directly to Qradar, but also write all to message files. In CustomLog setting, we push all to local1.info log facility, however in rsyslog setting we have a default setting to write down *.info to /var/log/message. I need to add local1.none in that statement to prevent write log to message file, then also prevent the whole web server from performance downgrade.

     

    You mean that you don't see any username information in the %u field when in fact there should be some information? I do not know, this is an Apache thing, maybe it has to do with the type of authentication. The Apache website seems to be down right now =) so I can't access the documentation.

  • networkingkool
    networkingkool
    12 Posts

    Re: Apache Access Logs

    ‏2014-06-20T10:34:54Z  
    • Alaa Ali
    • ‏2014-06-14T07:41:29Z

    You mean that you don't see any username information in the %u field when in fact there should be some information? I do not know, this is an Apache thing, maybe it has to do with the type of authentication. The Apache website seems to be down right now =) so I can't access the documentation.

    Hi Alaa Ali,

    I got big mistake about username field in Apache log. Our users are authenticated in application layer itself, not in Apache web layer. As a result we cannot get the authenticated user information from apache log.

    Thanks for comments.