Filebeat

API Gateway + API Data Store + Terracotta Server

In case, if you also want to aggregate Terracotta server logs that are stored in different location, perform the following steps:

If Terracotta Server and API Gateway is installed in the same machine, perform the following steps:

1. In the Administration > General > Application logs > Log aggregation section, disable the log aggregation.
   Note: By default, the Terracotta logs are available in the (user.home)/terracotta/server-logs location. You can change this folder by specifying the custom log data location in the tc-config.xml.
2. Update the filebeat_template.yml file in the SAG_Install_Dir\profiles\IS_Instance_Name\apigateway\filebeat location with the following content. Make sure you place the following content below the DashboardLogs configuration:

   - type: log
     # Change to true to enable this prospector configuration.
     enabled: true
     # Paths that should be crawled and fetched. Glob based paths.
     paths:
     - C:/Users/alice/terracotta/server-logs/terracotta-server.log
     fields:
       node: ${NODE}
       fileType: TerracottaServerLogs
       timezone: ${TIMEZONE}
     fields_under_root: true
     multiline.pattern: (([\s]+)20[0-9]{2}-)|20[0-9]{2}-
     multiline.negate: true
     multiline.match: after

3. In the Administration > General > Application logs > Log aggregation section, enable log aggregation.
4. In the Analytics > Application Logs section, check whether the Terracotta Server logs are aggregated.

If Terracotta Server and API Gateway are installed in different machine, perform the following steps:

1. Edit the tc-config.xml in Terracotta Server with the following content to store the logs in a network location and start the Terracotta Server:

   <?xml version="1.0" encoding="UTF-8"?>
   <tc:tc-config xmlns:tc="http://www.terracotta.org/config">
    
     <servers>
       <server host="localhost" name="%h">
         <data>C:/install/TerraCottaServer/Terracotta/server/bin/data</data>
         <logs>//worstation01/share/tcserverlog</logs>   
         <offheap>
           <enabled>true</enabled>
           <maxDataSize>512m</maxDataSize>
         </offheap>
       </server>
       <restartable enabled="true"/>
     </servers>
   </tc:tc-config>

2. In the Administration > General > Application logs > Log aggregation section, disable log aggregation.
3. Update the filebeat_template.yml file in the SAG_Install_Dir\profiles\IS_Instance_Name\apigateway\filebeat location with the following content. Make sure you place the following content below the DashboardLogs configuration:

   - type: log
     # Change to true to enable this prospector configuration.
     enabled: true
     # Paths that should be crawled and fetched. Glob based paths.
     paths:
     - //mckmut02/share/tc-server-logs/terracotta-server.log
     fields:
       node: ${NODE}
       fileType: TerracottaServerLogs
       timezone: ${TIMEZONE}
     fields_under_root: true
     multiline.pattern: (([\s]+)20[0-9]{2}-)|20[0-9]{2}-
     multiline.negate: true
     multiline.match: after

4. In the Administration > General > Application logs > Log aggregation section, enable the log aggregation.
5. In the Analytics > Application Logs section, check whether the Terracotta Server logs are aggregated.

Configuring Log for Elasticsearch Client in API Gateway

API Gateway uses Elasticsearch REST client to connect to API Data Store or External Elasticsearch. By Default, the logs that are created by these REST clients are ignored to avoid over logging. To troubleshoot or diagnose elasticsearch calls from API Gateway, you can manually configure the log configuration for Elasticsearch REST client using the following steps:

1. Open the log4j2.properties located at SAG_Install_Dir\profiles\IS_Instance_Name\configuration\logging.
2. Add the following configuration at the end of the file:

   logger.10.name=org.elasticsearch.client
   logger.10.additivity=false
   logger.10.level=info
   logger.10.appenderRef.lar.ref=ESRestClient
    
   logger.11.name=org.apache.http
   logger.11.additivity=false
   logger.11.level=info
   logger.11.appenderRef.lar.ref=ESRestClient
    
   logger.12.name=org.apache.http.wire
   logger.12.additivity=false
   logger.12.level=info
   logger.12.appenderRef.lar.ref=ESRestClient
    
   logger.13.name=org.apache.http.impl.conn
   logger.13.additivity=false
   logger.13.level=info
   logger.13.appenderRef.lar.ref=ESRestClient
    
   appender.esrestclient.name=ESRestClient
   appender.esrestclient.type=RollingFile
   appender.esrestclient.fileName=<INSTALL_LOCATION>/IntegrationServer/instances/default/logs/ESRestClient.log
   appender.esrestclient.filePattern=<INSTALL_LOCATION>/IntegrationServer/instances/default/logsESRestClient.log.%i
   appender.esrestclient.layout.type=PatternLayout
   appender.esrestclient.layout.pattern=%d [%t] %-5p %c %x - %m%n
   appender.esrestclient.policies.type=Policies
   appender.esrestclient.policies.size.type=SizeBasedTriggeringPolicy
   appender.esrestclient.policies.size.size=10MB
   appender.esrestclient.strategy.type=DefaultRolloverStrategy
   appender.esrestclient.strategy.max=10
   #This is a custom Platform provided filter which matches all log messages that contain OSGi data in MDC (bundle.id, component.name, etc.)
   appender.esrestclient.filter.osgi.type=LogServiceFilter
   appender.esrestclient.filter.osgi.onMatch=DENY
   appender.esrestclient.filter.osgi.onMismatch=NEUTRAL

3. Restart API Gateway.

   You can see the ESRestClient.log at the SAG_Install_Dir\IntegrationServer\instances\instance_name\logs location. You can see the request sent to particular elasticsearch node and what is the response status code in the ESRestClient.log