Interval probe manager

The interval probe manager provides probe points that fire at a user-defined time-interval. The probe points are not located in kernel or application code, but instead are based on wall clock time interval based probe events.

The interval probe manager is useful for summarizing statistics collected over an interval of time. It accepts a 4-tuple or a 5-tuple probe specification in the following format:

@@interval:*:clock:<# milliseconds>:[*|cpu_ids]

The interval probe manager will filter probe events by process ID if it is provided in the second field. Assigning the * to the second field indicates that the probe will be fired for all processes. Further, the only value supported by the interval probe manager for the third field is the clock keyword that identifies the probe specification as being for a wall clock probe. The fourth field, that is the <# milliseconds> field, identifies the number of milliseconds between firings of the probe. The interval probe manager requires that the value for this field consist only of digits 0-9.

The fifth field, <cpu_ids> identifies CPU IDs of the CPUs on which the probe occurs. The value of this field must be specified in the range of CPU IDs, or * or individual CPU IDs. The process ID specified in the second tuple and the CPU ID specified in the fifth tuple are mutually exclusive. Hence, the interval probe manager filters the probe events either by the specified process ID in tuple 2 running on any CPU or filters events for all the processes running on the specified CPUs. The minimum supported clock value is 100 milliseconds. Only one CPU bound interval probe per logical CPU in the system. This is an optional field.

For interval probes without process Id, intervals should be exactly divisible by 100. Thus, probe events that are apart by 100ms, 200ms, 300ms, and so on, are allowed in non-profiling interval probes. For interval probes with process Id specified, intervals should be greater or equal to minimum interval allowed for global root user or exactly divisible by 10 for other users. Thus, probe events that are apart by 10ms, 20ms, 30ms, and so on, are allowed for normal users in profiling interval probes. Only one profiling interval probe can be active for a process.

Examples for Interval probe specification for CPU bound probes are listed below:

The interval probe manager requires only basic dynamic tracing privileges. The interval probe manager enforces the following limits on the number of probes it supports to prevent malicious users from running the kernel out of memory by creating huge numbers of interval probes.

The interval probe manager does not support the following functions. If used inside an interval manager probe point, these functions will generate an empty string or zero as output.

• get_function
• get_probe
• get_location_point

When process ID is not specified, an interval probe can trigger in the context of any process depending upon when the probe fires since the probe event is based on wall clock time. Because of this, the ProbeVue framework does not allow the use of any of the following functions inside the interval probe manager's action block to prevent unauthorized access to a process's internal data. This security violation is caught only in the kernel. The Vue script will successfully compile but the session will fail to initialize.

• stktrace
• get_userstring

These functions provide no value when used from the probe manager. Even if you are the root user, you cannot call these functions inside the interval probe manager.

When the process ID is specified, the interval probe is triggered for all the threads within the process at the specified time interval. As the probe is fired in the context of the process, stktrace() function and __pname built-in is allowed inside the interval probe manager’s action block, unlike when process ID is not specified.

System trace probe manager

The system trace probe manager provides probe points wherever existing system trace hooks to trace channel zero (system event channel) occur, both within the kernel and within applications. To use this probe manager, you must have the kernel access privilege, and not be running in a WPAR.

The system trace probe manager accepts a 3-tuple probe specification in the following format:

@@systrace:*:<hookid>

where the hookid argument specifies the ID for the specific system trace hook of interest. The hookid argument consists of 4 hex digits typically of the form hhh0. For example, to specify the hookid argument for the fork system call, specify 1390. See the /usr/include/sys/trchkid.h file for examples, such as HKWD_SYSC_FORK. The entries in this file are hook words, where the hookid value is in the upper halfword. Because hook words can be arbitrary, no validation of the hookid argument beyond checking that it is a valid hex string of up to 4 hex digits is performed. It is not an error to specify a hookid value that never occurs.

As a convenience, you can specify the hookid argument with fewer than 4 hex digits. In this case, first a trailing zero is assumed, and then additional leading zeroes as necessary to implicitly define the required 4 digits. For example, you can use 139 as an abbreviation of 1390. Similarly, 0100, 010, and 10 all specify the same hookid value, taken from HKWD_USER1.

You can specify the hookid argument with the * wildcard character. This will probe all system tracing, with likely unacceptable performance implications. Hence, such a specification must be used only when absolutely necessary.

The second tuple is reserved, and must be specified as an asterisk, as shown.

Only system trace events that actually occur and record system trace data trigger probes. In particular, a system trace probe can only occur when system trace is active. The systrace probe manager is an event-based probe manager. Hence, probe name, function name, and location point are not available. As the hookword is passed to the script, this is not a significant restriction.

A non-root user is limited to at most 64 systrace probes simultaneously enabled. No more than 128 explicit systrace probes can be enabled system-wide.

ProbeVue built-in register variables allow access to the data traced. You cannot use the __arg* variables for this purpose. There are two general styles for system tracing.

The following style is for the trchook(64)/utrchook(64) (or the equivalent TRCHKLx macros in C) hooks:

• __r3 contains the 16 bit hookid.
• __r4 contains the subhookid.
• __r5 contains traced data word D1.
• __r6 contains traced data word D2.
• __r7 contains traced data word D3.
• __r8 contains traced data word D4.
• __r9 contains traced data word D5.

Not all trace hooks contain all 5 data words. Undefined data words from a given trace hook will appear as zero. The Vue clause for a given hook ID must know exactly what and how much data its hook ID traces.

If the trace record was produced by one of the functions in the trcgen or trcgent family, use the following style:

• __r3 contains the 16 bit hookid.
• __r4 contains the subhookid.
• __r5 contains traced data word D1.
• __r6 contains the length of the traced data.
• __r7 contains the address of the traced data.

The following script shows a simple example of the systrace probe manager:

	@@systrace:*:1390
	{
		if (__r4 == 0) {	/* normal fork is traced with subhookid zero */
			printf(“HKWD_SYSC_FORK: %d forks child %d\n”, __pid, __r5);
			exit();
		}
	}

Systrace probe manager is independent of the system trace function and can trace hook locations even if the system trace function is not in an active state. You can turn on the system trace function when a ProbeVue session is active.