Sampling policy

Sampling policy allows an Operations Dashboard administrator to define the percentage of traces that are sampled, collected and stored. Only the stored traces appear in Operations Dashboard Web Console, including dashboards, traces list, reports, alerts, etc.

By default, only ten percent (10%) of the traces is sampled. You may change this value (e.g. to increase it to 100% so all traces are sampled) via this page.

Sampling of less than 100% of the traces may be used to lower the resources required by Operations Dashboard to process and store the distributed tracing data. Keeping 100% of the traces enables investigation of every trace, but at the cost of higher resources. However, it may be argued that for investigating a major issue, even a sampling of 10% of recorded traces is sufficient, and saves 90% of the resources. Traces that are not sampled are not sent to Operations Dashboard, which means less tracing data network in the cluster and less CPU and storage space used by Operations Dashboard.

The decision whether to store a trace or not is made when the trace starts (by the tracing Agent at that capability). Once a trace is marked for sampling, this decision is stored in the trace context, and all spans of that trace will be sampled, to have a complete view of that sampled trace.