Sysproc probe manager

Process (or thread) creation or termination

Signal generation and delivery

Scheduler and dispatcher events

Scheduler and dispatcher dictate how a process or thread runs in the system. Administrator analyzes system performance by using dynamic trace scheduler or dispatcher subsystem.

The dynamic trace scheduler or dispatcher subsystem helps discover the reasons for retention of threads.

Dynamic Reconfiguration (DR), and CPU binding events

This class of probes offer dynamic tracing capabilities to a user who tracks resources bound to a process.

Probe specification

The following format must be used in a Vue script to probe sysproc events:

@@sysproc:<sysproc_event>:<pid/tid/*>

First tuple @@sysproc indicates that this probe is specific to sysproc events.

Second tuple specifies the event to be probed.

Probe points (events of interest)

A brief description of all events that can be probed through the sysproc probe manager is mentioned in the following table:

Method to access data at a probe-point

ProbeVue allows data access through built-in variables.

Following are the list of built values of type (1).

• __trcid
• __errno__kernelmode
• __arg1 to __arg7
• __curthread
• __curproc
• __mst
• __tid
• __pid
• __ppid
• __pgid
• __uid
• __euid
• __ublock
• __execname
• __pname

The built-in variables are also classified as context specific and context independent. Context-specific built-ins provide data based on the execution context of the probe.

AIX® kernel operates in thread or interrupt context. Context-specific probes produce correct result when probe is started at thread or process context.

Results that are obtained from context-specific built-ins in interrupt execution context might be unexpected. Context-independent built-ins do not depend on the execution context and can be accessed safely irrespective of probe execution environment.

Probe points

Probe points are the specific events for which a probe is fired. Following are the list of probe points.

forkfail

The forkfail probe starts when fork fails. This probe determines the reasons of fork failure.

Syntax: @@sysproc:forkfail:<pid/tid/*>

Special built-in supported

__forkfailinfo
{ 
fail_reason; 
}

The fail_reasonvariable has one of the following values:

Table 3. fail_reason probe: Failure reasons

Reason	Description
FAILED_RLIMIT	Failed due to rlimit limitations
FAILED_ALLOCATIONS	Failed due to internal resource allocations
FAILED_LOADER	Failed at a loader stage
FAILED_PROCDUP	Failed at procdup

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid, __ublock, __execname, __pname.

Execution environment

Runs in process environment.

Example

Following example shows how to monitor all fork failures because of rlimit in the system.

@@BEGIN 
{ 
         x = 0; 
} 

@@sysproc:forkfail:* 
        when (__forkfailinfo->fail_reason == FAILED_RLIMIT) 
{ 
                printf ("process %s with pid %llu failed to fork a child\n",__pname,__pid); 
                x++; 
} 
 
@@END 
{ 

        printf ("Found %d failures during this vue session\n",x); 
} 

execfail

The execfail probe starts when a exec function call fails. Use the execfail probe to determine the reasons for the failure.

Syntax: @@sysproc:execfail:<pid/tid/*>

Table 4. execfail probe: Failure reasons

Reason	Description
FAILED_PRIVILEGES	New process failed to acquire or inherit privileges
FAILED_COPYINSTR	New process failed to copy instruction
FAILED_V_USERACC	New process failed to discard v_useracc regions
FAILED_CLEARDATA	Failed during clearing data for new process
FAILED_PROCSEG	Failed to establish process private segment
FAILED_CH64	Failed to convert to a 64-bit process
FAILED_MEMATT	Failed to attach to a memory resource set
FAILED_SRAD	Failed to attach to a srad
FAILED_MSGBUF	Error message buffer length is zero
FAILED_ERRBUF	Failed to allocate error message buffer
FAILED_ENVAR	Failed to allocate environment variables
FAILED_CPYSTR	Copy string error
FAILED_ERRBUFCPY	Failed to copy the error messages from errmsg_buf
FAILED_TOOLNGENV	Env too long for allocated memory
FAILED_USRSTK	Failed to setup user stack
FAILED_CPYARG	Failed to copy arglist to stack
FAILED_INITPTRACE	Failed to init ptrace

Note: 64 is added to error value if loader error is encountered.

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid, __ublock, __execname, __pname.

Execution environment

Runs in process environment.

exit

This probe starts when a process exits. Exit is also a system call manager and is traced through system call probe manager. Probing exit system call through sysproc probe manager explains nature and reasons of exit. It also explains reasons for a user thread termination in kernel space and not returned to user space.

Syntax: @@sysproc:forkfail:<pid/tid/*>

A program can exit because of the following reasons:

• On reaching a terminal condition when a user space program cannot proceed further.
• On receiving a terminal signal.

Special built-in supported

__exitinfo{
	signo;	      	
	returnval;	    
	iscore;	     		
}

Where, signo value signifies the signal number that caused process termination, returnval is the value that is returned by exit. Nonzero signo is valid only if the program is stopped by a signal.

The iscore variable is set when a core is generated as a result of process exit.

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid, __ublock, __execname, __pname.

Execution Environment

Runs in process environment.

Example

Following example explains how to probe exit event

echo '@@sysproc:exit:* { printf (" %s    %llu    %llu\n", __pname, __pid,__exitinfo->returnval);}' | probevue

Which will produce an output similar to the following.

 ksh    5833042    0 
 telnetd    7405958    1 
 dumpctrl    7405960    0 
 setmaps    7275006    0 
 termdef    7274752    0 
 hostname    7274754    0 
 id    8257976    0 
 id    8257978    0 
 uname    8257980    0 
 expr    8257982    1 

threadcreate

threadcreate probe starts when a thread is created successfully.

Syntax: @@sysproc:threadcreate:<pid/tid/*>

Note: The specified pid or tid must be the process or thread ID of the process or thread that created the thread.

Special built-in supported

__threadcreateinfo
{
	tid;	
	pri;       
	policy; 
}

where tid indicates the thread id of new thread that is created, and priority is the priority of the thread. Policy denotes the thread scheduling policy of the thread.

Table 5. Policy values for the threadcreate probe

Policy	Description
SCHED_OTHER	default AIX scheduling policy
SCHED_FIFO	first in-first out scheduling policy
SCHED_RR	round robin scheduling policy
SCHED_LOCAL	local thread scope scheduling policy
SCHED_GLOBAL	global thread scope scheduling policy
SCHED_FIFO2	FIFO with RQHEAD after short sleep
SCHED_FIFO3	FIFO with RQHEAD all the time
SCHED_FIFO4	FIFO with weak preempt

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid, __ublock, __execname, __pname.

Execution environment

Runs in process environment (user or kproc).

Example

To continuously print all processes in the system creating a thread printing process name, creating process id , id of the newly created thread and creation time-stamp.

echo '@@sysproc:threadcreate:* 
{ printf ("%s %llu %llu %A\n",__pname,__pid,__threadcreateinfo->tid,timestamp());}' | probevue

An output similar to the following example is displayed.

nfssync_kproc 5439964 23921151 Feb/22/15 09:22:38 
nfssync_kproc 5439964 24052201 Feb/22/15 09:22:38 
nfssync_kproc 5439964 23920897 Feb/22/15 09:22:38 
nfssync_kproc 5439964 22479285 Feb/22/15 09:22:55 
nfssync_kproc 5439964 23920899 Feb/22/15 09:22:55 
nfssync_kproc 5439964 22479287 Feb/22/15 09:22:55

threadterminate

The probe strarts for a thread which is terminated.

Syntax: @@sysproc:threadterminate:<pid/tid/*>

Note: Specified process ID or thread ID must be corresponding to the process or thread currently getting stopped.

Special built-ins supported

None.

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

Runs in process environment (user or kproc).

Example

To continuously print all processes in the system terminating a thread printing process name, creating process id , id of the newly created thread and creation time-stamp.

# echo '@@sysproc:threadterminate:* { printf ("%s %llu %llu %A\n",__pname,__pid,__tid,timestamp());}' | probevue 

An output similar to the following sample is displayed:

nfssync_kproc 5439964 23855555 Feb/22/15 09:59:30 
nfssync_kproc 5439964 21758249 Feb/22/15 09:59:30 
nfssync_kproc 5439964 23855557 Feb/22/15 09:59:30

threadexcept

This probe starts when a program exception occurs. A program exception is generated when system detects a condition in which a program cannot continue normally. Some exceptions are fatal (illegal instruction) while some can be recovered (address space change).

Syntax: @@sysproc:threadexcept:<pid/tid/*>

Special built-ins supported

__threadexceptinfo
{
	pid;          	       
	tid;	       	       
	exception;	      	
	excpt_address        
}

where pid denotes process ID of the process that received exception, tid is the thread ID of the kernel thread that received exception, excpt_address is address that caused this exception while exception can assume one of the values as denoted in the table.

Table 6. Exception values for the threadexcept probe

Exception	Description
EXCEPT_FLOAT	Floating point exception
EXCEPT_INV_OP	Invalid op-code
EXCEPT_PRIV_OP	Privileged op in user mode
EXCEPT_TRAP	Trap instruction
EXCEPT_ALIGN	Code or data alignment
EXCEPT_INV_ADDR	Invalid address
EXCEPT_PROT	Protection
EXCEPT_IO	Synchronous I/O
EXCEPT_IO_IOCC	I/O exception from IOCC
EXCEPT_IO_SGA	I/O exception from SGA
EXCEPT_IO_SLA	I/O exception from SLA
EXCEPT_IO_SCU	I/O exception from SCU
EXCEPT_EOF	Reference beyond end-of-file (mmap)
EXCEPT_FLOAT_IMPRECISE	Imprecise floating point exception
EXCEPT_ESTALE_I	Stale text segment exception
EXCEPT_ESTALE_D	Stale data segment exception
EXCEPT_PT_WATCHP	Hit ptrace watchpoint

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

Runs in process or interrupt environment.

Note: Since this probe can start in an interrupt context, built-ins variables like __pid, __tid that depend upon the execution context might not indicate the process or thread id. Special built-in members for this probe guarantee correct process or thread id intended for the process or thread.

Example

Following example shows trace program exceptions generated by a prove event being traced by a debugger.

# cat threadexcept.e 
@@sysproc:threadexcept:* 
{ 
 printf ("PID = %llu TID= %llu EXCEPTION=%llu ADDRESS = %llu\n ",__threadexceptinfo->pid,__threadexceptinfo->tid,__threadexceptinfo-
>exception,__threadexceptinfo->excpt_address); 
} 

Run a debugging session on a program compiled with debugging support

# dbx a.out 
Type 'help' for help. 
Core file "core" is older than current program (ignored) 
reading symbolic information ... 
(dbx) stop in main 
[1] stop in main 
(dbx) r 
[1] stopped in main at line 5 
    5           int a=5;

An output similar to the following sample is displayed:

PID = 6816134 TID= 24052015 EXCEPTION=131 ADDRESS = 268436372

sendsig

This probe is started when a signal is sent to a process through external sources ( other process , process from user space, from kernel streams or Interrupt context)

Syntax:@@sysproc:sendsig:<pid/*>

__dispatchinfo{
	cpuid;		<- cpu id
	
	oldpid;            <- pid of the thread currently running
	oldtid;		<- thread id of the thread currently running
	oldpriority;	<- priority of the thread currenly running
	newpid;	<- pid of the new process process selected for running 
	newtid;	<- thread id of the thread selected for running
	newpriority;	<-priority of the thread selected for running 
}

where pid id the process identifier of the target process receiving the signal. This probe does not allow specifying a thread identifier to filter results specific to a thread.

Special built-ins

_sigsendinfo{
	tpid;               ← target pid
	spid;	 	← source pid  
	signo;	       ← signal sent
}

where tpid is the target source process identifier, spid identifies source of the signal. The spid is non-zero when signal is sent from user space or process context. Source process identifier is 0 if signal is sent from an exception or interrupt context. Signal number information is contained in signo.

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

Runs in process or interrupt environment.

Note: Since this probe can start in an interrupt context, it is possible that built-ins like __pid, __tid, which depend upon thread execution context might not indicate the process or thread id of interest. Special built-in members for this probe guarantee correct process or thread id intended for the process or thread.

When this probe starts in process context, built-in members that depend on execution context point to source process. built-in members like __pid, __tid, and __curthread provide information regarding the source process.

.

Example

To continuously print signal source signal target and signal number of all signals.

echo '@@sysproc:sendsig:* {printf ("Source=%llu Target=%llu sig=%llu\n",__sigsendinfo->spid,__sigsendinfo->tpid,__sigsendinfo->signo);}' | 
probevue

An output similar to the following sample is displayed:

Source=0 Target=6619618 sig=14 
Source=0 Target=8257944 sig=20 
Source=0 Target=8257944 sig=20

sigqueue

This probe starts when a queued signal is being sent to the process.

Syntax:@@sysproc:sigqueue:<pid/*>

Special built-ins

_sigsendinfo{
	tpid;               ← target pid
	spid;	 	← source pid  preprocess.cp
	signo;	       ← signal sent
}

Since posix signals are queued to a process, specifying thread identifier is not allowed in this probe.

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

This probe starts in the context of the sending process. Hence, context-based built-ins refer to the sending process in this probe event.

Execution environment

This probe runs in process context.

Example

echo '@@sysproc:sigqueue:*{printf ("%llu   %llu   %llu\n",__sigsendinfo->spid,__sigsendinfo->tpid,__sigsendinfo->signo);}' | probevue

An output similar to the following sample is displayed:

8258004   6095294   31
sigdispose
Syntax : @@sysproc:sigdispose:<pid/tid/*>

Probe starts when a signal is disposed to a target process. Specify process ID of the process which received this signal in the sysprobe specification to filter this probe.

Special built-ins

__sigdisposeinfo{
	tpid;         ← target pid
	ttid;          ← target tid 
	signo;      ← signal whose action is being taken.
	fatal;        ← will be set if the process is going to be killed as part of signal action
} 

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

This probe can start from process or interrupt context. If started from interrupt context, this probe might not provide required value for context-based built-ins.

Example

Continuously print process identifier, thread identifier, signal number and indicate if this signal disposal will result in termination of the process for all processes in the system.

cat sigdispose.e

@@sysproc:sigdispose:* 
{ 
 printf ("%llu  %llu %llu %llu\n",__sigdisposeinfo->tpid,__sigdisposeinfo->ttid, __sigdisposeinfo->signo,__sigdisposeinfo->fatal); 
} 

An output similar to the following sample is displayed:

5964064  20840935 14 0 
1  65539 14 0 
4719084  19530213 14 0

sigaction

Syntax:@@sysproc:sigaction:<pid/tid/*>

This probe starts when a signal handler is installed or replaced.

Special built-ins

__sigactioninfo{
	old_sighandle;            ← old signal handler function address
	new_sighandle;	←new signal handler function address 
	signo;		            ← Signal number 
	rpid;		            ← requester's pid
} 

old_sighandle will be 0 if a signal handler is installed for the first time.

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

This probe starts in process environment.

Note: AIX kernel ensures that only one signal is delivered to a process or thread at a time. Another signal to that process or thread is only sent when signal delivery is finished.

Example

To track the beginning and finish of all signals in a system:

@@sysproc:sighandlestart:* 
{ 
		 
		signal[__tid] = __sighandlestartinfo->signo; 
		printf ("Signal handler at address 0x%x invoked for thread id %llu to handle signal %llu\n",__sighandlestartinfo-
>sighandle,__curthread->tid,__sighandlestartinfo->signo); 
} 


@@sysproc:sighandlefinish:* 
{ 

		printf ("Signal handler completed for thread id %llu for  signal %llu\n",__curthread->tid,signal[__tid]); 
		delete (signal,__tid); 
}

An output similar to the following sample can be displayed:

Signal handler at address 0x20001d58 invoked for thread id 19923365 to handle signal 20 
Signal handler completed for thread id 19923365 for  signal 20 
Signal handler at address 0x10003400 invoked for thread id 20840935 to handle signal 14 
Signal handler completed for thread id 20840935 for  signal 14 
Signal handler at address 0x10002930 invoked for thread id 19530213 to handle signal 14 
Signal handler completed for thread id 19530213 for  signal 14 
Signal handler at address 0x300275d8 invoked for thread id 22348227 to handle signal 14 
Signal handler completed for thread id 22348227 for  signal 14 
Signal handler at address 0x20001a3c invoked for thread id 65539 to handle signal 14 
Signal handler completed for thread id 65539 for  signal 14

sighandlefinish

This probe starts at signal handler completion.

Syntax: @@sysproc:sighandlestart:<pid/tid/*>

Special built-ins supported: None.

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

Runs in process environment. Protected, context switch is not allowed on executing CPU.

changepriority

This probe starts when the priority of a process is being changed. This event is not a scheduler or dispatcher-enforced.

Syntax: @@sysproc:changepriority:<pid/tid/*>

Note: The priority change might also be unsuccessful; success of priority change is not guaranteed.

Special built-ins supported

__chpriorityinfo{
	pid;
	old_priority;   <- current priority 
	new_priority; <-  new scheduling priority of the thread.
}

Execution Environment

This probe runs in process environment.

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid, __ublock, __execname, __pname.

Example

To track all processes whose priority is being changed:

echo '@@sysproc:changepriority:* { printf ("%s priority changing from %llu to %llu\n",__pname,__chpriorityinfo->old_priority,__chpriorityinfo-
>new_priority);}' | probevue

An output similar to the following sample is displayed:

xmgc priority changing from 60 to 17 
xmgc priority changing from 17 to 60 
xmgc priority changing from 60 to 17 
xmgc priority changing from 17 to 60 
xmgc priority changing from 60 to 17

offreadyq

This probe starts when a thread is removed from a system run queue.

Syntax:@@sysproc:offreadyq:<pid/tid/*>

Special built-ins supported

__readyprocinfo{
	pid;		<- process id of thread becoming ready
	tid;		<- Thread id.
	priority;	<- priority of the thread
}

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

Runs in process or interrupt environment.

Use case: Trace time taken by a thread that is performing I/O operation to get back to ready queue.

@@BEGIN 
{ 
	printf ("           Pid      Tid      Time           Delta\n"); 
}   

@@sysproc:offreadyq :*
{ 
	ready[__tid] = timestamp(); 
	printf ("offreadyq: %llu %llu %W\n",__readyprocinfo->pid,__readyprocinfo->tid,ready[__tid]); 
} 

@@sysproc:onreadyq :*
{ 
	 
	if (diff_time(ready[__tid],0,MICROSECONDS)) 
	{ 
		auto:diff = diff_time (ready[__tid],timestamp(),MICROSECONDS); 
		printf ("onreadyq : %llu %llu %W      %llu\n",__readyprocinfo->pid,__readyprocinfo->tid,ready[__tid],diff); 
		delete (ready,__tid); 
	} 
}

An output similar to the following sample is displayed:

           Pid      Tid      Time           Delta 
offreadyq: 7799280 20709717 5s 679697µs 
onreadyq : 7799280 20709717 5s 679697µs      6 
offreadyq: 7799280 20709717 5s 908716µs 
onreadyq : 7799280 20709717 5s 908716µs      3 
offreadyq: 7799280 20709717 6s 680186µs 
onreadyq : 7799280 20709717 6s 680186µs      5 
offreadyq: 7799280 20709717 6s 710720µs 
onreadyq : 7799280 20709717 6s 710720µs      4 
offreadyq: 7799280 20709717 6s 800720µs 
onreadyq : 7799280 20709717 6s 800720µs      2 
offreadyq: 7799280 20709717 6s 882231µs 
onreadyq : 7799280 20709717 6s 882231µs      2 
offreadyq: 7799280 20709717 6s 962313µs 
onreadyq : 7799280 20709717 6s 962313µs      2 
offreadyq: 7799280 20709717 6s 980311µs 
onreadyq : 7799280 20709717 6s 980311µs      2 

onreadyq

This probe starts when a thread is enqueued to system ready queue or its position in ready queue is modified.

Syntax:@@sysproc:offreadyq:<pid/tid/*>

Special built-ins supported

__readyprocinfo{
	pid;		<- process id of thread becoming ready
	tid;		<- Thread id.
	priority;	<- priority of the thread
}

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

Runs in process or interrupt environment.

dispatch

This probe starts when system dispatcher is called to select a thread to run on a specific CPU.

Syntax:@@sysproc:dispatch:<pid/tid/*>

Special built-in supported

__dispatchinfo{
	cpuid;		<- CPU where selected  thread will run. 
	oldpid;            <- pid of the thread currently running
	oldtid;		<- thread id of the thread currently running
	oldpriority;	<- priority of the thread currenly running
	newpid;	<- pid of the new process process selected for running 
	newtid;	<- thread id of the thread selected for running
	newpriority;	<-priority of the thread selected for running 
}

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

Runs in interrupt environment only.

Example

print process thread id of old and selected thread on CPU '0' with  dispatch time relative to start of the script


echo '@@sysproc:dispatch:* when (__cpuid == 0){printf ("%llu %llu %W\n",__dispatchinfo->oldtid,__dispatchinfo->newtid,timestamp());}' | 
probevue

An output similar to the following example is displayed:

24641983 20709717 0s 48126µs 
20709717 23593357 0s 48164µs 
23593357 20709717 0s 48185µs 
20709717 23593357 0s 48214µs 
23593357 20709717 0s 48230µs 
20709717 23593357 0s 48288µs 
23593357 261 0s 48303µs 
261 20709717 0s 48399µs

Example II

Time spent on  CPU '0' by threads in between dispatch event.

@@BEGIN 
{ 
	printf ("Thread cpu Time-Spent\n"); 
} 

@@sysproc:dispatch:* when (__cpuid == $1) 
{ 
	if (savetime[__cpuid] != 0) 
		auto:diff = diff_time (savetime[__cpuid],timestamp(),MICROSECONDS); 
	else 
		diff = 0; 
	savetime[__cpuid] = timestamp(); 
	printf ("%llu %llu %llu\n",__dispatchinfo->oldtid,__dispatchinfo->cpuid,diff); 
}		 
	
# probevue cputime.e 6 
Thread cpu Time-Spent 
3146085 6 0 
3146085 6 9995 
3146085 6 10002 
3146085 6 10008 
3146085 6 99988 
3146085 6 100006 
3146085 6 99995 
3146085 6 99989 
3146085 6 100010 
3146085 6 100001 
3146085 6 100000 
3146085 6 99998

As can be observed thread 3146085 is being re-dispatched on the CPU at an interval of 1sec in absence of any other thread competing for this CPU.

oncpu

This probe starts when a new process or thread acquires CPU.

Syntax:@@sysproc:oncpu:<pid/tid/*>

Where pid is process identifier and tid is thread identifier of process or thread that is acquiring the CPU.

Special built-ins supported

__dispatchinfo{
	cpuid;		<- CPU where selected  thread will run. 
	newpid;	<- pid of the new process process selected for running 
	newtid;	<- thread id of the thread selected for running
	newpriority;	<-priority of the thread selected for running 
}

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

Runs in interrupt environment only.

Example

To print time spent by threads of sysncd on all CPU's
#!/usr/bin/probevue

@@BEGIN

{

	printf ("PROCESSID THREADID CPU TIME\n");

}



@@sysproc:oncpu:$1 

{

	savetime[__cpuid] = timestamp();

}



@@sysproc:offcpu:$1  

{

	if (savetime[__cpuid] != 0)

		auto:diff = diff_time (savetime[__cpuid],timestamp(),MICROSECONDS);

	else

		diff = 0;

	printf ("%llu %llu %llu %llu\n",

		__dispatchinfo->oldpid,

		__dispatchinfo->oldtid,

		__dispatchinfo->cpuid,

		diff);

}


# cputime.e `ps aux|grep syncd| grep -v grep| cut -f 6 -d " "`

An output similar to the following example is displayed:

3735998 18612541 0 2
3735998 15663427 0 1
3735998 15073557 0 1
3735998 18743617 0 1
3735998 18874693 0 1
3735998 18809155 0 15
3735998 18940231 0 20
3735998 18547003 0 1
3735998 19267921 0 1
3735998 19071307 0 17
3735998 18678079 0 1
3735998 18481465 0 1
3735998 19202383 0 15
3735998 19005769 0 1
3735998 19136845 0 19
3735998 6160689 0 190

offcpu

This probe starts when a process or thread is dispatched from a CPU.

Syntax:@@sysproc:dispatch:<pid/tid/*>

Special built-ins supported

__dispatchinfo{
	cpuid;		<- CPU where selected  thread will run. 
	newpid;	<- pid of the new process process selected for running 
	newtid;	<- thread id of the thread selected for running
	newpriority;	<-priority of the thread selected for running 
}

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

Runs in interrupt environment only.

blockthread

This probe starts when a thread is blocked from running on a CPU. Blocking is a form of sleeping when a thread sleeps without holding any resources.

Syntax: @@sysproc:blockthread:*

Special built-ins supported

__sleepinfo{
	pid;
	tid;
	waitchan;   <-- wait channel of this sleep.	
}

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

Runs in interrupt environment only.

foldcpu

This probe starts when a CPU core is about to be folded. This probe does not happen in process context and must not be filtered with a pid or tid.

Syntax: @@sysproc:foldcpu:*

Special built-ins supported

__foldcpuinfo{
	cpuid;		<- logical cpu id which triggers core folding 
	gpcores;         <- general purpose (unfolded, non-exclusive) cores available.
}

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7.

Example:

To track all CPU folding events in the system:

__foldcpuinfo{
	cpuid;		<- logical cpu id which triggers core folding 
	gpcores;         <- general purpose (unfolded, non-exclusive) cores available.
}

bindprocessor

Syntax: @@sysproc:bindprocessor:<pid/tid/*>

This probe starts when a thread or process is bound to a CPU. Bindprocessor is a permanent event and must not be confused with temporary CPU switches.

Special built-ins supported

__bindprocessorinfo{
	ispid       <- 1 if cpu is bound to process; 0 for a thread 
	id;	    <- thread or process id.
	
	cpuid;
	
};

Other supported built-ins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

Runs in process environment.

changecpu

This probe starts when a thread changes CPU temporarily. This event is more likely to be captured during CPU funneling events or intentional jumps of some kproc events to perform CPU related tasks (the xmgc process jumps to all CPUs to manage kernel heaps) special built-ins.

Syntax: @@sysproc:changecpu:*>

Special built-ins supported

__changecpuinfo
{
	oldcpuid;	<-source CPU
	newcpuid; 	<- target CPU   
	pid;
	tid;		<-Thread id
}

Other supported builtins

__errno__kernelmode, __arg1 to __arg7, __curthread, __curproc, __mst, __tid, __pid, __ppid, __pgid, __uid, __euid.

Execution environment

Runs in process environment.

Example

@@sysproc:changecpu:*

{
printf ("changecpu PID=%llu TID=%llu old_cpuid=%d new_cpuid= %d \n",
__changecpuinfo->pid,__changecpuinfo->tid,__changecpuinfo->oldcpuid,__changecpuinfo->newcpuid);

}

An output similar to the following example is displayed:

changecpu PID=852254 TID=1769787 old_cpuid=26 new_cpuid= 27 

changecpu PID=852254 TID=1769787 old_cpuid=-1 new_cpuid= 0 

changecpu PID=852254 TID=1769787 old_cpuid=0 new_cpuid= 1 

changecpu PID=852254 TID=1769787 old_cpuid=1 new_cpuid= 2

resourceattach

This probe is fired when a resource is attached to another resource in the system.

Syntax: @@sysproc:resourceattach:*>

Special built-ins supported

__srcresourceinfo{
	type;
	subtype;
	id;		<- resource type identifier
	offset; 		<-offset if a memory resource
	length;		<- length if a memory resource
	policy;
}
__tgtresourceinfo{
	type;
	subtype;
	id;		<- resource type identifier
	offset;		<-offset if a memory resource
	length;		<- length if a memory resource
	policy;		
}

Where type and subtype could be have one of the following values.

Table 7. The resourceattach probe: type and subtype values

Resource type	Description
R_NADA	Nothing - invalid specification
R_PROCESS	Process
R_RSET	Resource set
R_SUBRANGE	Memory range
R_SHM	Shared Memory
R_FILDES	File identified by an open file
R_THREAD	Thread
R_SRADID	SRAD identifier
R_PROCMEM	Process Memory

Other supported builtins

__errno__kernelmode, __arg1 to __arg7, __mst.

Execution environment

Runs in process environment.

resourcedetach

This probe is fired when a resource is detached from another resource in the system.

Syntax: @@sysproc:resourcedetach:*>

Special built-ins supported

__srcresourceinfo{
	type;
	subtype;
	id;		<- resource type identifier
	offset; 		<-offset if a memory resource
	length;		<- length if a memory resource
	policy;
}

__tgtresourceinfo{
	type;
	subtype;
	id;		<- resource type identifier
	offset;		<-offset if a memory resource
	length;		<- length if a memory resource
	policy;		
}

Where type and subtype could be have one of the following values.

Table 8. The resourcedetach probe: type and subtype values

Resource type	Description
R_NADA	Nothing - invalid specification
R_PROCESS	Process
R_RSET	Resource set
R_SUBRANGE	Memory range
R_SHM	Shared Memory
R_FILDES	File identified by an open file
R_THREAD	Thread
R_SRADID	SRAD identifier
R_PROCMEM	Process Memory

Other supported builtins

__errno__kernelmode, __arg1 to __arg7, __mst, __tid, __pname.

Execution environment

Runs in process environment.

drphasestart

This probe is fired when a dr handler is about to be called.

Syntax: @@sysproc:drphasestart:*

Special built-ins supported

__drphaseinfo{
	dr_operation;   ← dr operation
	dr_flags;	
	dr_phase;
	handler_rc;    ← always 0 in drphasestart
}

dr_operation can have one of the following values:

• DR operation
• DR_RM_MEM_OPER
• DR_ADD_MEM_OPER
• DR_RM_CPU_OPER
• DR_ADD_CPU_OPER
• DR_CPU_SPARE_OPER
• DR_RM_CAP_OPER
• DR_ADD_CAP_OPER
• DR_RM_RESMEM_OPER
• DR_PMIG_OPER
• DR_WMIG_OPER
• DR_WMIG_CHECKPOINT_OPER
• DR_WMIG_RESTART_OPER
• DR_SOFT_RES_CHANGES_OPER
• DR_ADD_MEM_CAP_OPER
• DR_RM_MEM_CAP_OPER
• DR_CPU_AFFINITY_REFRESH_OPER
• DR_AME_FACTOR_OPER
• DR_PHIB_OPER
• DR_ACC_OPER
• DR_CHLMB_OPER
• DR_ADD_RESMEM_OPER

dr flags can be a combination of the following values:

• Flag
• DRP_FORCE
• DRP_RPDP
• DRP_DOIT_SUCCESS
• DRP_PRE_REGISTERED
• DRP_CPU DRP_MEM DRP_SPARE
• DRP_ENT_CAP
• DRP_VAR_WGT
• DRP_RESERVE
• DRP_PMIG DRP_WMIG
• DRP_WMIG_CHECKPOINT
• DRP_WMIG_RESTART
• DRP_SOFT_RES_CHANGES
• DRP_MEM_ENT_CAP
• DRP_MEM_VAR_WGT
• DRP_CPU_AFFINITY_REFRESH
• DRP_AME_FACTOR
• DRP_PHIB
• DRP_ACC_UPDATE
• DRP_CHLMB

Other supported builtins

__errno__kernelmode, __arg1 to __arg7, __tid

Execution environment

Runs in process or interrupt environment.

Example