Analyzing logs

Filtering and grouping logs

You can filter and group logs by using the following approaches:

Query builder
Directly from within a log
Filter sidebar

You can use each of the approaches individually, but the best results are achieved when you combine these approaches.

Using a query builder

To filter and group logs with the query builder, complete the following steps:

In the Filter field on the Analytics logs dashboard, click Add filter.
From the drop-down list, select the required filter option. A constructor appears on the filtering area with an equals operator as default.

Note: You can filter logs by using any attribute, such as level, message, stream, custom tags, snapshot, trace IDs, and exceptions. To filter logs by using regular expressions, see the supported regular expression dialect in the ClickHouse documentation.
Enter a relevant value in the input field of the equals operator. For standard values or related Instana entities, select values from the drop-down list.

Figure 1. Filtering and grouping
To troubleshoot problems, filter by using the is present operator to make a quick query on exceptions. You can change the operator by clicking it.

Figure 2. Filtering and grouping
To add more than one filter, an AND boolean operator appears as default.
To remove any of the filtering or the operators, click the filter or operator and click the x symbol.

Figure 3. Filtering and grouping
To apply grouping, click Add group, and select one of the tags. A common use case for grouping is to find out which services or hosts are generating more logs, which can help scope down the search.

Figure 4. Filtering and grouping

In this example, you can focus on the group with the service name spans-service by clicking the Focus on this group icon near the Number of logs data. Instana adds the service name as a filter and removes the grouping by service name.

Note: The Number of logs values might be approximate. When you apply no filter or only common filters, Instana does not query all data to get the exact number of logs. Instead, it uses prepared data to improve UI responsiveness.

Note: If you filter and group by a tag that is internally represented as a collection (for example, Host > fqdn), and one entry of the collection matches the filter, Instana lists all other entries from the collection as individual groups.

Directly from within a log

When the log message contains custom tags, they are highlighted in gray.

To filter directly from within the log message, complete the following steps:

Identify custom tags in the log message. The custom tags are highlighted in gray.
To add the custom tags as filter from within the log message, click the highlighted custom tag, and select Add as filter.

In the following example, you can see the remote address 10.255.201.71 and the remote host 10.255.201.71 are custom tags.

Figure 5. Filtering from log message

The following image shows the result of adding the custom tags as filter from within the log message.

Figure 6. Custom tag in query builder
To view detailed information that is related to the log, expand the log. The log tag table is displayed.
Hover over each row to find contextual actions.
Use the following contextual actions within a log to execute the actions and display a log list with the source log.
- Group by Tag icon: Displays a view where all the logs for the specific time frame and filters are grouped by that specific tag and the different values that it takes. This view is convenient to get hints of log volume for a specific tag.
  
  Note: To add a specific tag and value as a filter and enable endless scrolling, click the focus on this group icon. If you expand a group, only an overview is provided and endless scrolling is not available in this step.
  
  Figure 7. Group by tag
- Add tag as filter icon: Adds the tag and its value as a filter. The default is added as AND.
  
  Figure 8. Add tag as filter
  
  In the following example, the stream is added to the former applied filter WARN.
  
  Figure 9. Result tag added as filter
- Copy to clipboard icon: Copies the tag value to the clipboard. The tag value can be used when you create a troubleshooting ticket for your team.
  
  Figure 10. Copy to clipboard
To get information on an entity's health, check the health indicator before the entity name as shown in the following image. The color codes relate with the Smart Alerts color-code. Hover over it to see the number of issues for that entity, which gives a hint on the magnitude of a specific problem without changing the context.

Figure 11. Entities health

From side filter bar

The side filter bar provides a flexible way of filtering and grouping in combination with the query builder and directly from within a log.

To filter and group with the side filter bar, use the following tags:
- Log levels
- Stream
- Services
To view the number of logs for each different value of that tag even before you apply any filters, look at the number that is displayed near the category. In the following example, you can see that 30.3k logs are provided for the error tag.
To add a value to the filter, select the checkbox for the value.
To remove the value, clear the checkbox for the value.

You can also group by a filter directly from the icon near the main category title.

Saving filters

To store the filters that you applied to your data view, use the Save filters option. You can reuse these filters later or share them with other users in your tenant.

After applying the filters, click Save. The Save filters window is displayed.
In the Name field, enter a name for the filter. The name can include up to 75 characters.
If you want to include the selected group in the saved filter, select Include group.
Click Save to store the filter.

The saved filter is now available for reuse and can be accessed by all users in the same tenant.

You can edit or delete the filters. To edit a filter, click the edit icon and then update the filter name or included group. To delete a filter, click the delete icon.

Sharing information with your team

You can work with logs that are needed to share information with other members of your team. By clicking the link icon on a specific log, you can share a short link with your team. When anyone else uses the link, Instana shows the same screen with the time frame, filters, and source log that are highlighted, open, and centered in the screen, to facilitate collaboration.

Viewing logs in real time

You can view logs in real time on the Logs page. You can filter the logs, search for a keyword in the logs, and reuse filters across sessions and users.

To display only the logs that meet your criteria as described in Filtering and grouping, use the Add filter option. To search for specific text or keywords in your logs, enter a word or phrase in the Search bar. The console highlights matching entries immediately. To follow logs as they appear, set Tailing to "ON" when tailing is active, the console automatically scrolls to the latest logs as they are added. You can pause tailing at any time to review earlier logs by setting Tailing to "OFF". To save your active filters, click Save as described in Saving filters.