Filtering with dynamic focus

Dynamic Focus is our robust search and filter function that is capable of searching through all of data contexts simultaneously. Microservice architectures utilize a complex variety of data contexts that are all clustered together under one hood. Dynamic Focus is capable of filtering across these to provide a strong system-wide understanding for the user.

This cross-contextualization exposes powerful queries, for example:

  • Filter infrastructure
    • with open issues
    • of a specific technology
    • for a specific service and change event
  • Filter events
    • for a specific service and infrastructure combination
    • on a specific container and technology
    • on specific infrastructure

This feature takes advantage of our unique architecture and capabilities:

  • Dynamic Graph – Our cohesive model of physical and logical components, including all historical attributes and metrics. The Dynamic Graph also includes distributed traces and events. All nodes in the graph are connected based on their dependencies. The various views in Instana visualize the data of the graph, and Dynamic Focus actually filters the graph to provide specific subviews and focus.

  • Auto Discovery – All components, and their dependencies, held in the graph are auto-discovered in realtime by our Agent. Any saved filter will adapt to the current situation of your systems and applications without any manual change or configuration.

  • Incidents – Our Incidents are based on KPIs and machine learning and provide condensed information about an incident. This includes the dependencies to services, traces, and physical components, and they can be used by our Dynamic Focus to filter only the components that are part of a particular Incident.

Focus on application components

Here is an example to help you understand the concept of Dynamic Focus. We take the graph representation of a zoned application that is running on two hosts with four containers each. There is a JVM running in every container, and each of the two hosts runs two Tomcats and two Elasticsearch instances which provide a service per host. A simplified version of our Dynamic Graph model looks like this:

Our new filtering capability allows us to reduce the nodes in the graph based on a search query.

A query for entity.type:tomcat would match all Tomcat instances in the graph. This means that containers with Elasticsearch entities and the corresponding services would no longer be visible when viewing your infrastructure, application, and traces in our standard views.

Any chart, change, issue, or incident in Instana would then only be seen in the context of the reduced Graph.

Another example query is to search for all JVMs with 1.8 version; entity.jvm.version:1.8.*

In this case, the graph looks like this:

This is a powerful tool to find and focus user attention on single entities, or groups of entities, within the Graph that match certain criteria.

Focus on management attributes

Instana also has structures that are outside of the Dynamic Graph, but do reference it, for example, Events (like a Change), Issues, or Incidents. An Incident could contain services where Instana found KPI outliers, as well as containing entities with issues like high CPU load.

Using the new search capability, you can query and filter for Incidents like event.state:open event.type:incident. This filters all active Incidents in the currently selected timeframe. In this example, we assume there is only one active Incident when a degradation of an Elasticsearch service causes subsequent degradation of a Tomcat service.

As the Incident that matches will have references to the two services, the query will filter to only the relevant entities.

This also works the other way around. If we search for a Service, then only the Incidents that contain this Service would be shown.

Syntax

Instana uses the Lucene query syntax to build complex queries out of terms mapped to application components as well as boolean operators. From out of this, a robust variety of query types are supported, like free text searches, tag filtering, CPU count queries, and more. The search bar aides the user in building queries by automatically highlighting valid, individual terms out from the rest of the string. This helps avoid common string syntax pitfalls such as confusing the free text query "and" with the boolean operator AND.

The form that Lucene queries take varies depending on the implementation. In Instana, all queries proceed from a field to a field modifier to a quantifier. Both field values are separated by a period, while the quantifier is set off by a colon. For example, say a user wanted to filter out all Tomcat instances in their application. Their query in the search bar would read: entity.type: tomcat. This tells Instana to remove everything from view except for all entities of the type labelled "tomcat."

Of course entity.type:tomcat is just one query phrase. Different phrases can be strung together with boolean operators such as AND, OR, and NOT to build flexible and complex filters that show the client exactly what they need. An example:

Default operator

The AND operator is the default conjunction operator. This means that if there is no Boolean operator between two terms, the AND operator is used.

For example: entity.type:tomcat event.state:open is equivalent to entity.type:tomcat AND event.state:open.

It's worth noting that this diverges from the Lucene standard, which uses OR by default.

Complex sub-queries

While the Lucene query syntax and Dynamic Focus Queries look like they support complex boolean terms, this is not the case.

Especially the negation of sub-queries doesn't work as expected in Dynamic Focus Query.

entity.kubernetes.namespace:my-namespace AND NOT (entity.kubernetes.pod.name:my-name AND event.text:"ContainersNotReady")

The above query will not yield the expected results but be processed in a way that the terms in the sub-query inside the parentheses are not associated to each other as they would when applying boolean logic and the result would resemble the following query:

entity.kubernetes.namespace:my-namespace AND NOT entity.kubernetes.pod.name:my-name AND NOT event.text:"ContainersNotReady"

Auto-complete

In addition to highlighting Lucene query terms, Instana's search bar is also capable of IDE-grade proposal for uncompleted, searchable values. This is a tremendous aid to both new and experienced Instana users. Those who do not have a command of Lucene syntax can get automatic help, while experienced users can execute complicated searches much faster. All query proposals appear in a dropdown box as follows the search bar. If the needed proposal is in view, simply click or navigate to it with the arrow keys.

Instana's search function is zero-time, which means results appear as you enter in your search criteria. You are also able to save, edit, and delete filters to speed up your workflow.

Known limitations

Syntax constraints

  • White space is not allowed between the ! and the query to be negated. Use !<query> or NOT <query> instead.
  • When multiple of the same entity.application.name, entity.service.name, entity.endpoint.name Dynamic Focus keywords are used together, they can only be joined by the OR operator. E.g. the queries entity.application.name:foo AND entity.application.name:bar, entity.application.name:foo entity.application.name:bar are not supported. The query entity.application.name:foo OR entity.application.name:bar is supported.
  • When multiple of different entity.application.name, entity.service.name, entity.endpoint.name Dynamic Focus keywords are used together, they can only be joined by the AND operator. E.g. the query entity.application.name:foo OR entity.service.name:bar is not supported, whereas the query entity.application.name:foo AND entity.service.name:bar is supported.
  • Queries containing a negation of application, service or endpoint name are not supported. Consequently, neither NOT entity.application.name:foo, !entity.service.name:bar nor NOT entity.endpoint.name:foobar are valid. However, filtering out events of an application, service or endpoint is still possible by using NOT event.entity.label:foo instead.
  • Dynamic Focus queries entity.application.name:<term>, entity.service.name:<term> and entity.endpoint.name:<term> are evaluated using the string contains logic, e.g. Dynamic Focus Query entity.application.name:shop would also match entity.application.name:shop-frontend.
  • Dynamic Focus query keywords entity.application.name, entity.service.name and entity.endpoint.name are only supported in boolean queries (see boolean logic constraints above), term queries, phrase queries and prefix queries.

Query limitations

  • Dynamic Focus query on event.text containing hyphens might return unexpected results. For e.g. event.text:"foo-bar-random" would also return events containing just foo or bar or random due to tokenization of the Lucene analyzer.
  • Entities as follows a Docker container, such as Host or Availability Zone, cannot be selected when scoping on most kubernetes keywords. As an example, the query entity.kubernetes.deployment.name:my-K8s-deployment AND entity.selfType:host does not return any host. In contrast, the query entity.kubernetes.deployment.name:my-K8s-deployment AND entity.selfType:docker returns all docker containers related to the deployment. However, there are a few kubernetes keywords that can be used to select Host entites, nameley entity.kubernetes.cluster.* and entity.kubernetes.node.*. For instance, the query entity.kubernetes.cluster.label:my-K8s-cluster AND entity.selfType:host returns all related hosts of the cluster.

FAQ

Using wildcard (*) inside double quotes ("") does not yield expected results.

No matches are returned since the * symbol is evaluated as a character and not as a wildcard in this case. According to the lucene specification single and multiple character wildcard searches in phrase queries are not supported. A Phrase query is a group of words surrounded by double quotes, such as event.text:"something bad happened".

I am getting search result counts which seem inconsistent.

Sometimes, the result counts may be confusing. For instance, when searching within the infrastructure map, keep in mind that searches within views can match more entities than are actually visible within views, such as services which are not visible in the infrastructure map.

How do I know what quantifiers to use after the field and field modifiers?

Any measurable metric can be a field quantifier. For entities it would be the name, for events it would be the classification. Also, Instana's autocomplete functions can help with offering valid inputs for each part of the query.

What fields are searched when I do not specify any fields in the search query?

When not specifying any fields in a search query, Dynamic Focus will target the current context to provide meaningful results. This means it will search entity fields when on the infrastructure map, event data when on the event view, and so on.

What is the difference between entity.selfType and entity.type?

Searching for entity.selfType will limit the search results to the searched for entity, while entity.type will include all children of the searched for entity. For example, when you search for entity.selfType:host, the infrastructure map will only show the host entities (and all parent entities, like the zone). But when you search for entity.type:host, the search results will also include what is running on the host, e.g. webservers, databases, etc. However, both entity.type and entity.selfType yield the same results for entity types of the Application Perspectives. As an example, when using the query entity.type:endpoint in the Events view, the list will only contain events that have been detected on endpoint entities.