Running ProbeVue

Dynamic tracing is only allowed for users with privileges or for the superuser.

Authorizations and privileges

This is unlike the static tracing facilities in AIX®, which enforce relatively limited privilege checking. There is a reason for requiring privileges to run the probevue command. A Vue script can potentially produce more severe impacts on system performance than a static tracing facility like AIX system trace. This is because probe points for system trace are pre-defined and restricted. ProbeVue can potentially support many more probe points and the probe locations can potentially be defined almost anywhere. Further, ProbeVue trace actions at a probe point can take much longer to issue than the system trace actions at a probe point since those are limited to explicit data capture.

In addition, ProbeVue allows you to trace processes and read kernel global variables, both of which need to be controlled to prevent security exposures. A ProbeVue session can also consume a lot of pinned memory and restricting usage of ProbeVue to users with privilege reduces the risk of denial of service attacks. ProbeVue also allows administrators to control the memory usage of ProbeVue sessions through the SMIT interface.

Privileges for dynamic tracing are obtained differently depending upon whether role-based access control (RBAC) is enabled or not. Please refer to the AIX man pages for more information about enabling and disabling RBAC.

Note that in legacy or RBAC-disabled mode, there are no authorizations. Regular users cannot acquire privileges to run the probevue command to start a dynamic tracing session or run the probevctrl command to administer ProbeVue. Only the superuser can have privileges for both these functions. Do not disable RBAC when using ProbeVue unless you prefer to restrict this facility to root users only.

RBAC-enabled mode

Privileges in an RBAC system are obtained through authorizations. An authorization is a text string associated with security-related functions or commands. Authorizations provide the mechanism to grant rights to you to perform privileged actions. Only a user with sufficient authorization can issue the probevue command and start a dynamic tracing session.

aix.ras.probevue.trace.user.self

This authorization allows you to trace their applications in user space. The user ID of the process to be traced must be equal to the real user ID of the user invoking the probevue command. This authorization allows you to enable probe points provided by the uft probe manager for your processes. However, the effective, real and saved user IDs of the process to be traced must be equal. Thus, you cannot trace setuid programs with just this authorization.

aix.ras.probevue.trace.user

This authorization allows you to trace any application in user space including setuid programs and applications started by the superuser. Be careful when handing out this authorization. This authorization allows you to issue the probevue command and enable probe points provided by the uft probe manager for any process on the system.

aix.ras.probevue.trace.syscall.self

This authorization allows you to trace system calls made by their applications. The effective, real and saved user IDs of the process making the system call must be the same and equal to the real user ID of the user invoking the probevue command. This authorization allows you to enable probe points provided by the syscall probe manager for your processes. The second field of the probe specification must indicate the process ID for a process started by you.

aix.ras.probevue.trace.syscall

This authorization allows you to trace system calls made by any application on the system including setuid programs and applications started by the superuser. Be careful when handing out this authorization. This authorization allows you to issue the probevue command and enable probe points provided by the syscall probe manager for any process. The second field of the probe specification can either be set to a process ID to probe a specific process or to * to probe all processes.

aix.ras.probevue.trace

This authorization allows you to trace the entire system and includes all the authorizations defined in the preceding sections. You can also access and read kernel variables when running the probevue command, as well as to trace system trace events using the systrace probe manager. Be careful when handing out this authorization.

aix.ras.probevue.manage

This authorization allows you to administer ProbeVue. This includes changing the values of the different ProbeVue parameters, starting or stopping ProbeVue and viewing details of dynamic tracing sessions of all users when running the probevctrl command. Without this authorization, you can use the probevctrl command to view session data for dynamic tracing sessions started by you or view the current values for ProbeVue parameters.

aix.ras.probevue.rase

This authorization allows you to access to a highly privileged set of "RAS events" Vue functions which can produce system and LMT trace records, create live dumps, and even lead to the system abend. This privilege must be very carefully controlled.

aix.ras.probevue

This authorization grants all dynamic tracing privileges and is equivalent to all the preceding authorizations combined.

The superuser (or root) has all these authorizations assigned by default. Other users will need to have authorizations assigned to them by first creating a role with a set of authorizations and assigning the role to the user. The user will also need to switch roles to a role that has the required authorizations defined for dynamic tracing before invoking the probevue command. The following script is an example of how to provide user "joe" authorization to enable user space and system call probes for processes started by "joe".

 	mkrole authorizations=
   "aix.ras.probevue.trace.user.self,aix.ras.probevue.trace.syscall.self" 
   apptrace
	chuser roles=apptrace joe
	setkst -t role		# Copy roles to kernel (Or wait until system reboots)

User "joe" can be set up to always have all roles acquired by default when logging in or can switch to the role as needed using the following command:

swrole apptrace

ProbeVue privileges

The privileges that are available for ProbeVue are listed in the following table. A description of each privilege and the authorizations that map to that privilege is provided. Privileges form a hierarchy where the parent privilege contains all of the rights that are associated with the privileges of its children, but it can include additional privileges also.

ProbeVue parameters

All ProbeVue parameters can be modified through the SMIT interface (use the "smit probevue" fast path) or directly through the probevctrl command. ProbeVue can be stopped if there are no active dynamic tracing sessions and it can be restarted after stopping it without requiring a reboot. ProbeVue can fail to stop if any sessions that used thread-local variables had been previously active.

The following table summarizes the parameters defined for dynamic tracing sessions. In the description, a privileged user refers to the superuser or a user with the aix.ras.probevue.trace authorization and a non-privileged user is one who does not have this authorization.

Profiling ProbeVue Session

The ProbeVue framework provides a profiling facility that can be turned on or off to estimate the impact of enabled probes on the application. This facility accumulates the time taken by probe actions when they are started and reports when requested or when the session ends.

The profiling report displays the probe string and the time taken by the action corresponding to that probe string. The time that is consumed by the probe action is maintained as a list where the data collected is total, minimum, maximum, and average time taken by probe action. Profiling data also displays number of times that the probe action was timed. When you are looking up the profile for multiple functions through one probe string (by using regular expression or * in place of function name), profiling data provides an accumulated data of probes started for all such functions. It does not provide timing details for functions that are probed separately but only per-probe action.

The BEGIN and END probe actions are not profiled with this facility. These profiling details are session-specific details. You can enable probevue session profiling along with session start by using the probevue command or probevctrl command.

For more information, see the probevue and probevctrl commands.

Sample programs

Example 1

The following canonical "Hello World" program prints "Hello World" into the trace buffer and exits:

#!/usr/bin/probevue
	
	/* Hello World in probevue */
	/* Program name: hello.e */
	
	@@BEGIN
	{
		printf("Hello World\n");
		exit();
	}

Example 2

The following "Hello World" program prints "Hello World" when you types Ctrl-C on the keyboard:

#!/usr/bin/probevue
	
	/* Hello World 2 in probevue */
	/* Program name: hello2.e */
	
	@@END
	{
		printf("Hello World\n");
	}

Example 3

The following program shows how to use thread-local variables. This Vue script counts the number of bytes written to a particular file. It assumes that the processes are single-threaded or those threads that open files are the same ones that write to them. It also assumes that all write operations are successful. The script can be terminated at any time and you can obtain the current count of bytes written by typing Ctrl-C on the terminal.

#!/usr/bin/probevue
	
	/* Program name: countbytes.e */
	int open( char * Path, int OFlag, int mode );
	int write( int fd, char * buf, int sz);
	int done;

	@@syscall:*:open:entry
		when (done != 0 )
	{
		if (get_userstring(__arg1, -1) == "/tmp/foo") {
			thread:trace = 1;	
			done = 1;
		}
	}
	
	@@syscall:*:open:exit
		when (thread:trace)
	{
		thread:fd = __rv;
	}
	
	@@syscall:*:write:entry
		when (thread:trace && __arg1 == thread:fd)
	{
		bytes += __arg3;	/* number of bytes is third arg */ 
	}
	
	@@END
	{
		printf("Bytes written = %d\n", bytes);	
	}

Example 4

The following tentative tracing program shows how to trace the arguments passed to the read system call only if it returns zero bytes when reading the foo.data file:

#!/usr/bin/probevue
	/* File: ttrace.e */
	/* Example of tentative tracing */
	/* Capture parameters to read system call only if read fails */
	int open ( char* Path, int OFlag , int mode );
	int read ( int fd, char * buf, int sz);
	
	@@syscall:*:open:entry
	{
		filename = get_userstring(__arg1, -1);
		if (filename == "foo.data") {
			thread:open = 1;
			start_tentative("read");
			printf("File foo.data opened\n");
		}
	}
	
	@@syscall:*:open:exit
		when (thread:open == 1)
	{
		  thread:fd = __rv;
		  start_tentative("read");
		  printf("fd = %d\n", thread:fd);
		  thread:open = 0;
	}
		 
	@@syscall:*:read:entry
		when (__arg1 == thread:fd)
	{
		start_tentative("read");
		printf("Read fd = %d, input buffer = 0x%08x, bytes = %d,",
			__arg1, __arg2, __arg3);
		end_tentative("read");
		thread:read = 1;
	}
	
	@@syscall:*:read:exit
		when (thread:read == 1)
	{
		if (__rv < 0) {
			/* The printf below, even though non-tentative, is only
			 * executed in error cases and merges with the 
			 * previously printed tentative data
			 */
			printf(" errno = %d\n", __errno);
			commit_tentative("read");
		}
		else
			discard_tentative("read");
		thread:read = 0;
	}

A possible output if the read failed because a bad address (say 0x1000) was passed as input buffer pointer could look like the following output:

#probevue ttrace.e
File foo.data opened
fd = 4
Read fd = 4, input buffer = 0x00001000, bytes = 256, errno = 14

Example 5

The following Vue script prints the values of some kernel variables and exits immediately. Pay attention to the exit function in the @@BEGIN probe:

/* File: kernel.e */
/* Example of accessing kernel variables */
/* System configuration structure from /usr/include/sys/systemcfg.h */
struct system_configuration {
	int architecture;	/* processor architecture */
	int implementation;	/* processor implementation */
	int version;		/* processor version */
	int width;		/* width (32 || 64) */
	int ncpus;		/* 1 = UP, n = n-way MP */
	int cache_attrib;	/* L1 cache attributes (bit flags)	*/
				/* bit		0/1 meaning		*/
				/* -------------------------------------*/
				/* 31	 no cache / cache present	*/
				/* 30	 separate I and D / combined    */
	int icache_size;	/* size of L1 instruction cache */
	int dcache_size;	/* size of L1 data cache */
	int icache_asc;		/* L1 instruction cache associativity */
	int dcache_asc;		/* L1 data cache associativity */
	int icache_block;	/* L1 instruction cache block size */
	int dcache_block;	/* L1 data cache block size */
	int icache_line;	/* L1 instruction cache line size */
	int dcache_line;	/* L1 data cache line size */
	int L2_cache_size;	/* size of L2 cache, 0 = No L2 cache */
	int L2_cache_asc;	/* L2 cache associativity */
	int tlb_attrib;		/* TLB attributes (bit flags)		*/
				/* bit		0/1 meaning		*/
				/* -------------------------------------*/
				/* 31	 no TLB / TLB present		*/
				/* 30	 separate I and D / combined    */
	int itlb_size;		/* entries in instruction TLB */
	int dtlb_size;		/* entries in data TLB */
	int itlb_asc;		/* instruction tlb associativity */
	int dtlb_asc;		/* data tlb associativity */
	int resv_size;		/* size of reservation */
	int priv_lck_cnt;	/* spin lock count in supevisor mode */
	int prob_lck_cnt;	/* spin lock count in problem state */
	int rtc_type;		/* RTC type */
	int virt_alias;		/* 1 if hardware aliasing is supported */
	int cach_cong;		/* number of page bits for cache synonym */
	int model_arch;		/* used by system for model determination */
	int model_impl;		/* used by system for model determination */
	int Xint;		/* used by system for time base conversion */
	int Xfrac;		/* used by system for time base conversion */
	int kernel;		/* kernel attributes			    */
				/* bit		0/1 meaning		    */
				/* -----------------------------------------*/
				/* 31	32-bit kernel / 64-bit kernel	    */
                                /* 30   non-LPAR      / LPAR                */
                                /* 29   old 64bit ABI / 64bit Large ABI     */
                                /* 28   non-NUMA      / NUMA                */
                                /* 27   UP            / MP                  */
                                /* 26   no DR CPU add / DR CPU add support  */
                                /* 25   no DR CPU rm  / DR CPU rm  support  */
                                /* 24   no DR MEM add / DR MEM add support  */
                                /* 23   no DR MEM rm  / DR MEM rm  support  */
                                /* 22   kernel keys disabled / enabled	    */
                                /* 21   no recovery   / recovery enabled    */
                                /* 20   non-MLS    / MLS enabled	    */
	long long physmem;	/* bytes of OS available memory		    */
	int slb_attr;		/* SLB attributes			    */
				/* bit		0/1 meaning		    */
				/* -----------------------------------------*/
				/* 31		Software Managed	    */
	int slb_size;		/* size of slb (0 = no slb)		    */
	int original_ncpus;	/* original number of CPUs		    */
	int max_ncpus;		/* max cpus supported by this AIX image     */
	long long maxrealaddr;	/* max supported real memory address +1     */
	long long original_entitled_capacity;
				/* configured entitled processor capacity   */
				/* at boot required by cross-partition LPAR */
				/* tools.				    */
	long long entitled_capacity; /* entitled processor capacity	    */
	long long dispatch_wheel; /* Dispatch wheel time period (TB units)  */
	int capacity_increment;	/* delta by which capacity can change	    */
	int variable_capacity_weight;	/* priority weight for idle capacity*/
					/* distribution			    */
	int splpar_status;	/* State of SPLPAR enablement		    */
				/*	0x1 => 1=SPLPAR capable; 0=not	    */
				/*	0x2 => SPLPAR enabled 0=dedicated;   */
				/*			     1=shared       */
	int smt_status;		/* State of SMT enablement                  */
				/*    0x1 = SMT Capable  0=no/1=yes         */
				/*    0x2 = SMT Enabled  0=no/1=yes         */
				/*    0x4 = SMT threads bound true 0=no/1=yes */
	int smt_threads;	/* Number of SMT Threads per Physical CPU   */
        int vmx_version;        /* RPA defined VMX version, 0=none/disabled */
	long long sys_lmbsize;	/* Size of an LMB on this system. */
	int num_xcpus;		/* Number of exclusive cpus on line */
	signed char errchecklevel;/* Kernel error checking level */
	char pad[3];		/* pad to word boundary		*/
        int dfp_version;        /* RPA defined DFP version, 0=none/disabled */
				/* if MSbit is set, DFP is emulated         */
};

__kernel struct system_configuration _system_configuration;

@@BEGIN
{
	String s[40];
	int j;
	__kernel int max_sdl;	/* Atomic RAD system decomposition level */
	__kernel long lbolt;	/* Ticks since boot  */

	printf("No. of online CPUs\t\t= %d\n", _system_configuration.ncpus);

	/* Print SMT status */
	printf("SMT status\t\t\t=");
	if (_system_configuration.smt_status == 0)
		printf(" None");
	else {
		if (_system_configuration.smt_status & 0x01)
			printf(" Capable");
		if (_system_configuration.smt_status & 0x02)
			printf(" Enabled");
		if (_system_configuration.smt_status & 0x04)
			printf(" BoundThreads");
	}
	printf("\n");

	/* Print error checking level */
	if (_system_configuration.errchecklevel == 1)
		s = "Minimal";
	else if (_system_configuration.errchecklevel == 3)
		s = "Normal";
	else if (_system_configuration.errchecklevel == 7)
		s = "Detail";
	else if (_system_configuration.errchecklevel == 9)
		s = "Maximal";
	printf("Error checking level\t\t= %s\n",s);

	printf("Atomic RAD system detail level\t= %d\n", max_sdl);

	/* Long in the kernel is 64-bit, so we use %lld below */
	printf("Number of ticks since boot\t= %lld\n", lbolt);

	exit();
}

The following output is a possible output when you run the preceding script on a Power 5 dedicated partition with default kernel attributes:

# probevue kernel.e
No. of online CPUs              = 4
SMT status                      = Capable Enabled BoundThreads
Error checking level            = Normal
Atomic RAD system detail level  = 2
Number of ticks since boot      = 34855934

Probe managers

The probe manager is not part of the basic ProbeVue framework, but is, nevertheless, an essential component of dynamic tracing. Probe Managers are the providers of the probe points that can be instrumented by ProbeVue.

Probe managers generally support a set of probe points that belong to some common domain and share some common feature or attribute that distinguishes them from other probe points. Probe points are useful at points where control flow changes significantly, at points of state change or at other points of significant interest. Probe managers are careful to select probe points only in locations that are safe to instrument.

Probe managers can choose to define their own distinct rules for the probe specifications within the common style that must be followed for all probe specifications.

ProbeVue supports the following probe managers:

• System call probe manager
• User function probe manager
• Interval probe manager
• System trace probe manager
• Extended System Call probe manager
• I/O probe manager
• Network probe manager
• Sysproc probe manager

System call probe manager

The syscall probe manager supports probes at the entry and exit of well-defined and documented base AIX system calls. These are the system calls that have the same interface at the libc.a (or C library) entry point and in the kernel entry point. Either the system call is a pass-through (the C library simply imports the symbol from the kernel and the exports it with no code in the library) or there is trivial code for the interface inside the library.

The syscall probe manager accepts a 4-tuple probe specification in one of the following formats:

• syscall:*:<system_call_name>:entry

• syscall:*:<system_call_name>:exit

Additionally, the syscall probe manager also accepts a 4-tuple probe specification in one of the following formats:

• syscall:<process_ID>:<system_call_name>:entry

• syscall:<process_ID>:<system_call_name>:exit

where a process ID can be specified as the second field of the probe specification to support probing of specific processes.

The system call names accepted by the syscall probe manager are the names of the libc.a interfaces and not the kernel's internal system call names. For example, the read subroutine is exported by libc.a, but the actual system call name or kernel entry point is kread. The syscall probe manager will internally translate a libc interface to its kernel entry point and enable the probe at entry into the kread kernel routine. Because of this, if multiple C library interfaces invoke the kread routine, the probe pointfires for those interfaces also. Generally, this is not a problem because for most of the system calls supported by the syscall probe manager, there is a 1-to-1 mapping between the libc interface and the kernel routine.

For each syscall probe, there is an equivalent probe point in the library code provided by the uft probe manager. The uft probe manager does support all library interfaces (unless it is a passthrough interface and there is no code for the call or references to it in the library at all) including those not supported by the syscall probe manager. However, the syscall probe manager has two advantages:

• The syscall probe manager can probe every process in the system by specifying asterisk as the second field.
• The syscall probe manager is more efficient than the uft probe manager because it does not need to switch from user mode to kernel mode and back to run the probe actions.

For more information about the full list of system calls supported by the syscall probe manager see ProbeVue.

UFT probe manager

The uft or the user function tracing probe manager supports probing user space functions that are visible in the XCOFF symbol table of a process. The uft probe manager supports probe points that are at entry and exit points of functions whose source is a C or FORTRAN language text file even though the symbol table can contain symbols whose sources are from a language other than C or FORTRAN.

The tracing of Java™ applications in a way identical to the existing tracing mechanism from the users point of view and the JVM is one that performs most of the real tasks on behalf of Probevue.

For probing java application see "Java Applications Probe Manager" below.

The uft probe manager accepts a 5-tuple probe specification in the following format:

uft:<processID>:*:<function_name>:<entry|exit>

When the third field is set to *, the UFT probe manager searches the function in all of the modules loaded into the process address space including the main executable and shared modules. This implies that if a program contains more than one C function with this name (for example, functions with static class that are contained in different object modules), then probes will be applied to the entry point of every one of these functions.

If a function name in a specific module needs to be probed, the module name needs to be specified in the third field. The probe specification syntax to provide the library module name is illustrated below:

# Function foo in any module
@@uft:<pid>:*:foo:entry 
# Function foo in any module in any archive named libc.a
@@uft:<pid>:libc.a:foo:entry 
# Function foo in the shr.o module in any archive named libc.a
@@uft:<pid>:libc.a(shr.o):foo:entry	  

The function name in the fourth tuple can be specified as an Extended Regular Expression (ERE). The ERE should be enclosed between "/ and /" like "/<ERE>/".

 /* Probe entry of all libc.a functions starting with “malloc” word */
@@uft:$__CPID:libc.a: “/^malloc.*/”:entry      
/* Probe exit of all functions in the executable a.out */
@@uft:$__CPID:a.out:”/.*/”:exit

In the entry probes, where a function name is specified as a regular expression, individual arguments cannot be accessed. However, probevue function print_args can be used to print the function name and its arguments. The argument values is printed based on the argument type information available in the traceback table of the function.

In the exit probes, where a function name is specified as a regular expression, return value cannot be accessed.

Probevue supports enabling probes in more than one process at the same time. However, you will need privileges even for probing processes that belong to you.

Probevue enforces a restriction that prevents processes with user-space probes from being debugged using the ptrace or procfs based APIs.

As indicated above, the uft probe manager supports probes in shared modules like shared library modules. The following script shows an example that traces mutex activity by enabling probes in the thread library's mutex lock and unlock subroutines.

/* pthreadlocks.e */
/* Trace pthread mutex activity for a given multithreaded process */
/* The following defines are from /usr/include/sys/types.h */

typedef long long pid_t;
typedef long long thread_t;

typedef struct {
	int	__pt_mutexattr_status;	
	int	__pt_mutexattr_pshared;	
	int	__pt_mutexattr_type;
} pthread_mutexattr_t;

typedef struct __thrq_elt thrq_elt_t;

struct __thrq_elt {
	thrq_elt_t	*__thrq_next;
	thrq_elt_t	*__thrq_prev;
};

typedef volatile unsigned char _simplelock_t;

typedef struct __lwp_mutex {
	char		__wanted;
	_simplelock_t	__lock;
} lwp_mutex_t;

typedef struct {
	lwp_mutex_t		__m_lmutex;
	lwp_mutex_t		__m_sync_lock;
	int			__m_type;
	thrq_elt_t		__m_sleepq;
	int			__filler[2];
} mutex_t;

typedef struct {
	mutex_t			__pt_mutex_mutex;
	pid_t			__pt_mutex_pid;
	thread_t		__pt_mutex_owner;
	int			__pt_mutex_depth;
	pthread_mutexattr_t	__pt_mutex_attr;
} pthread_mutex_t;

int pthread_mutex_lock(pthread_mutex_t *mutex);
int pthread_mutex_unlock(pthread_mutex_t *mutex);

@@uft:$__CPID:*:pthread_mutex_lock:entry
{
	printf("thread %d: mutex 0x%08x locked\n", __tid, __arg1);
}

@@uft:$__CPID:*:pthread_mutex_unlock:entry
{
	printf("thread %d: mutex 0x%08x unlocked\n", __tid, __arg1);
}

The probe specification, argument access and ProbeVue functions usage in probe actions for Fortran function probes is similar to other uft probes with the following differences:

• User has to map the Fortran data types to ProbeVue data types and use the same in the script. The mapping of Fortran basic data types to ProbeVue data types is listed in the below table.

  Table 3. Fortran to ProveVue data types mapping

  Fortran data-type	ProbeVue data-type
  INTEGER * 2	short
  INTEGER * 4	int/long
  INTEGER * 8	long long
  REAL	float
  DOUBLE PRECISION	double
  COMPLEX	No equivalent basic data type. This needs to be mapped to a structure as shown below: typedef struct complex { float a; float b; } COMPLEX;
  LOGICAL	int (The Fortran standard requires logical variables to be the same size as INTEGER/REAL variables)
  CHARACTER	char
  BYTE	signed char

• Fortran passes IN scalar arguments of internal procedures by value, and other arguments by reference. Arguments passed by reference should be accessed with copy_userdata(). More information on argument association in fortran can be found in the Argument association topic.
• Routine names in a Fortran program is case in-sensitive. But, while specifying them in a ProbeVue script, they should be in lower-case .
  The following sample script illustrates how to map Fortran data types to ProbeVue data types:

  /* cmp_calc.e */
  /* Trace fortran routines 
  cmp_calc(COMPLEX, INTEGER) and 
  cmplxd(void) */
  
  typedef struct complex{
          float a;
          float b;
          } COMPLEX;
  
  typedef int INTEGER;
  
  /* arguments are indicated to be of pointer type as they are passed by reference */
  void cmp_calc(COMPLEX *, INTEGER *);  
  void cmplxd();
  
  @@uft:$__CPID:*:cmplxd:entry
  {
  printf("In cmplxd entry \n");
  }
  
  @@uft:$__CPID:*:cmp_calc:entry
  {
  COMPLEX c;
  int i;
  copy_userdata(__arg1, c);
  copy_userdata(__arg2, i);
  printf("%10.7f+j%9.7f  %d \n", c.a,c.b,i);
  }

• Fortran stores arrays in column-major form, whereas ProbeVue stores in row-major form and the below script shows how users can retrieve the array elements.

  /* array.e*/
  /* ProbeVue script to probe fortran program array.f */
  
  void displayarray(int **, int, int);
  @@uft:$__CPID:*:displayarray:entry
  {
  int a[5][4];		/* row and column sizes are interchanged */
  copy_userdata(__arg1, a);
  /* to print the first row */
  printf("%d %d %d \n”, a[0][0], a[1][0], a[2][0]);
  /* to print the second row */
  printf(“%d %d %d\n", a[0][1], a[1][1], a[2][1]);
  }
  
  /* Fortran program array.f */
  
  PROGRAM ARRAY_PGM
  IMPLICIT NONE 
  INTEGER, DIMENSION(1:4,1:5) :: Array 
  INTEGER :: RowSize, ColumnSize
  CALL ReadArray(Array, RowSize, ColumnSize) 
  CALL DisplayArray(Array, RowSize, ColumnSize) 
  CONTAINS
  SUBROUTINE ReadArray(Array, Rows, Columns)
  IMPLICIT NONE
  INTEGER, DIMENSION(1:,1:), INTENT(OUT) :: Array
  INTEGER, INTENT(OUT) :: Rows, Columns
  INTEGER :: i, j
  READ(*,*) Rows, Columns
  DO i = 1, Rows
  READ(*,*) (Array(i,j), j=1, Columns)
  END DO
  END SUBROUTINE ReadArray
  SUBROUTINE DisplayArray(Array, Rows, Columns) 
  IMPLICIT NONE 
  INTEGER, DIMENSION(1:,1:), INTENT(IN) :: Array
  INTEGER, INTENT(IN) :: Rows, Columns 
  INTEGER :: i, j 
  DO i = 1, Rows 
  WRITE(*,*) (Array(i,j), j=1, Columns )
  END DO 
  END SUBROUTINE DisplayArray
  END PROGRAM ARRAY_PGM

• Intrinsic or built-in functions cannot be probed with ProbeVue . All FORTRAN routines as listed in the XCOFF symbol table of the executable/linked libraries can be probed. ProbeVue uses the XCOFF symbol table to identify the location of these routines. However, the prototype for the routine has to be provided by the user and ProbeVue tries to access the arguments according to the prototype provided. For routines where the compiler mangles the routine names, the mangled name should be provided. Since Vue is a C-style language, user should ensure that the FORTRAN function/subroutine prototype is appropriately mapped to C language style function prototype. Please refer to the linkage conventions for argument passing and function return values in the Passing data from one language to another topic. The below example illustrates this:

  /* Fortran program ext_op.f */
  /* Operator “*” is extended for rational multiplication */
  MODULE rational_arithmetic
  IMPLICIT NONE
          TYPE RATNUM
                  INTEGER :: num, den
          END TYPE RATNUM
          INTERFACE OPERATOR (*)
                  MODULE PROCEDURE rat_rat, int_rat, rat_int
          END INTERFACE
          CONTAINS
          FUNCTION rat_rat(l,r)      ! rat * rat
                  TYPE(RATNUM), INTENT(IN) :: l,r
                  TYPE(RATNUM) :: val,rat_rat
                  val.num=l.num*r.num
                  val.den=l.den*r.den
                  rat_rat=val
          END FUNCTION rat_rat
          FUNCTION int_rat(l,r)      ! int * rat
                  INTEGER, INTENT(IN)      :: l
                  TYPE(RATNUM), INTENT(IN) :: r
                  TYPE(RATNUM) :: val,int_rat
                  val.num=l*r.num
                  val.den=r.den
                  int_rat=val
          END FUNCTION int_rat
          FUNCTION rat_int(l,r)      ! rat * int
                  TYPE(RATNUM), INTENT(IN) :: l
                  INTEGER, INTENT(IN)      :: r
                  TYPE(RATNUM) :: val,rat_int
                  val.num=l.num*r
                  val.den=l.den
                  rat_int=val
          END FUNCTION rat_int
  END MODULE rational_arithmetic
  PROGRAM Main1
  Use rational_arithmetic
  IMPLICIT NONE
  	TYPE(RATNUM) :: l,r,l1
  	l.num=10
          l.den=11
          r.num=3
          r.den=4
          L1=l*r
  END PROGRAM Main1
  
  /* ext_op.e */
  /* ProbeVue script to probe routine that gets called when “*”
      is used to multiply rational numbers in ext_op.f */
  
  struct rat
  {
          int num;
          int den;
  };
  struct rat rat;
  void __rational_arithmetic_NMOD_rat_rat(struct rat*,
  	struct rat*,struct rat*); 
  /* Note that the mangled function name is provided. */    
  /* Also, the structure to be returned is sent in the buffer whose address is provided as the first argument. */
  /* The first explicit parameter is in the second argument. */
  @@BEGIN
  {
          struct rat* rat3;
  }
  @@uft:$__CPID:*:__rational_arithmetic_NMOD_rat_rat:entry
  {
  	struct rat rat1,rat2;
          copy_userdata((struct rat *)__arg2,rat1);
          copy_userdata((struct rat *)__arg3,rat2);
          rat3=__arg1;
  	/* The address of the buffer where the returned structure will be stored is saved at the function entry */
          printf("Argument Passed rat_rat = %d:%d,%d:%d\n",rat1.num,rat1.den,rat2.num,rat2.den);
  }
  @@uft:$__CPID:*:__rational_arithmetic_NMOD_rat_rat:exit
  {
          struct rat rrat;
          copy_userdata((struct rat *)rat3,rrat);	
          /* The saved buffer address is used to fetch the returned structure */
          printf("Return from rat_rat = %d:%d\n",rrat.num,rrat.den);
          exit();
  }

• ProbeVue won’t support direct inclusion of Fortran header files in the script. However, a mapping of Fortran data types to ProbeVue data types can be provided in a ProbeVue header file and included with the “-I’’ option.

C++ applications probe manager

C++ Probe Manager supports probing of C++ applications in a way identical to C probe managers. Support for "uft" style entry/exit probes on any C++ function, including member, overloaded, operator, and template functions in the core executable. A function entry/exit probe in C++ must use the @@uftxlc++ probe manager.

All tuples in the @@uftxlc++ style probe specifications have the same usage and format as for the @@uft style probe strings, with the exception of the function name. Because C++ allows a single function name to be overloaded, the function name specified in the probe string may have to include the function's argument types to uniquely identify the function being probed.

For example:

@@uftxlc++:12345:*:"foobar(int, char *)":entry

@@uftxlc++:12345:*:void foobar<int>(int, char *):entry

When probing a class member function or a function defined in a namespace, the fully qualified function name must be used in the probe string. To avoid any ambiguity between the single colon (:) tuple separator in probe strings and the double colon (::) scope resolution operator in a fully qualified C++ name, the entire function name tuple in the probe string must be quoted.

@@uftxlc++:12345:*:"Foo::bar(int)":entry

Example:

Below is c++ application

#include  "header.cc"
main()
{
int             i = 10;
incr_num(i);
float           a = 3.14;
incr_num(a);
char            ch = 'A';
incr_num(ch);
double          d = 1.11;
incr_num(d);
}

Content of the "header.cc"

# cat header.cc
#include <iostream.h>
template <class T>
T incr_num( T a)
{
return (++a);
}
int dummy()
{
int  i=10,j=20;
incr_num(i);
float a=3.14;
incr_num(a);
char  ch ='A',dh='Z';
incr_num(ch);
double d=1.1,e=1.11;
incr_num(d);
return  0;
}

Content of the Vue script vue_cpp.e

##C++
#include "header.cc"
##Vue
@@uftxlc++:$__CPID:*:"incr_num<int>(int)":entry
{
printf("Hello1_%d\n",__arg1 );
}
@@uftxlc++:$__CPID:*:"incr_num <  float   > (float)" :entry
{
printf("Hello2_%f\n",__arg1 );
}
@@uftxlc++:$__CPID:*:"incr_num <  char    > ( char )":entry
{
printf("Hello3_%c\n",__arg1 );
}
@@uftxlc++:$__CPID:*:"incr_num <  double    > ( double )":entry
{
printf("Hello4_%lf\n",__arg1 );
exit();
}

Execution :

/usr/vacpp/bin/xlC  app.c++
#  probevue  -X ./a.out  vue_cpp.e
Hello1_10
Hello2_3.140000
Hello3_A
Hello4_1.110000

/* Probe entry of all the C++ functions in the executable a.out */
@@uftxlc++:$__CPID:a.out:”/.*/”:entry
/* Probe exit of all the C++ functions with ‘foo’ word in it */
@@uftxlc++:$__CPID:*:”/foo/”:exit

In the entry probes, where a function name is specified as a regular expression, individual arguments cannot be accessed. However, probevue function print_args() can be used to print the function name and its arguments. The argument values is printed based on the argument type information available in the traceback table of the function.

In the exit probes, where a function name is specified as a regular expression, return value cannot be accessed.

Java applications probe manager

Java Probe Manager (JPM) supports probing of Java applications in a way identical to C and C++ probe managers. A single Vue script should be able to trace multiple java applications at the same time by using different process IDs of the JVMs. The same script can be used to probe syscalls or C/C++ applications along with Java applications and can use other probe managers.

Like uft (user function tracing) probe manager java probe manager also accepts 5-tuple probe specification in the following format:

uftjava :< process_ID> :*:< _qualified_function_name >: entry

Where the second tuple is the process ID of JVM process corresponding to the Java application that is being traced.

Third field: reserved for future use.

Fourth field: where the java method needs to be specified.

This name is a completely qualified name as used in java applications like Mypackage.Myclass.Mymethod.

Some of the restrictions that may apply are

• Only pure java methods can be probed, Native (shared library calls) or encrypted codes are not traceable.
• Only entry probes are supported.
• Can support only JVM v 1.5 and above that supports JVMTI interface.
• At any given point of time, no two Probevue sessions can probe the same Java application with @@uftjava.
• Polymorphic/Overloaded methods are not supported.
• Tracing/accessing external variables with same name as any of the Probevue keywords or built-in names are not supported. This may need those external symbols (Java application variable names) to be renamed.
• Accessing arrays of java applications is not supported in this release.
• Accessing arrays of java applications is not supported in this release.
• get_function () built-in for java language is not supported in this release.

Data Access: The action blocks of java probes can access the following data similar to existing behavior.

• Action block can access global, local and kernel script variables.
• Action block can access method arguments (Entry class variables) of primitive types.
• Action block can access the built-in variables.
• Action block can access Java application variables through fully qualified names, only static (class members).

  x = some_package.app.class.var_x;    //Access static/class member.

• Accessing java application primitive types variables is supported; they must be converted/promoted/casted implicitly without losing value to equivalent types in Vue language. But the actual memory usage (size) may differ from that of Java language.

The functions supported in the context of Java probe manager are listed in the following table:

Changes to Probevue command:

For Example:

probevue -X /usr/java5/bin/java -A  -agentlib:probevuejava myjavaapp  myscript.e

When running the 64 bit JVM, we have to use "agentlib:probevuejava64" as in:

probevue -X /usr/java5_64/bin/java -A  -agentlib:probevuejava64 myjavaapp  myscript.e 
where myjavaapp is the java class of myjavaapp.java application

Example ExtendedClass.java Source:

class BaseClass
{
        static int i=10;

        public static void test(int x)
        {
                i += x;
        }
}

public class ExtendedClass extends BaseClass
{
        public static void test(int x, String msg)
        {
                i += x;
                System.out.print("Java: " + msg + "\n\n");
                BaseClass.test(x);
        }

        public static void main(String[] args)
        {
                BaseClass.test(5);
                ExtendedClass.test(10, "hello");
        }
}

Example test.e script for above Java application:

@@uftjava:$__CPID:*:"BaseClass.test":entry
{
        printf("BaseClass.i: %d\n", BaseClass.i);
        printf("BaseClass.test: %d\n", __arg1);
        stktrace(0, -1);
        printf("\n");
}

@@uftjava:$__CPID:*:"ExtendedClass.test":entry
{
        printf("BaseClass.i: %d\n", BaseClass.i);
        printf("ExtendedClass.test: %d, %s\n", __arg1, __arg2);
        stktrace(0, -1);
        printf("\n");
}

Example ProbeVue session with above script:

# probevue -X /usr/java5/jre/bin/java \
-A "-agentlib:probevuejava ExtendedClass" test.e
Java: hello

BaseClass.i: 10
BaseClass.test: 5
BaseClass.test()+0
ExtendedClass.main()+1

BaseClass.i: 15
ExtendedClass.test: 10, hello
ExtendedClass.test()+0
ExtendedClass.main()+8

BaseClass.i: 25
BaseClass.test: 10
BaseClass.test()+0
ExtendedClass.test()+39
ExtendedClass.main()+8

Interval probe manager

The interval probe manager provides probe points that fire at a user-defined time-interval. The probe points are not located in kernel or application code, but instead are based on wall clock time interval based probe events.

The interval probe manager is useful for summarizing statistics collected over an interval of time. It accepts a 4-tuple probe specification in the following format:

@@interval:*:clock:<# milliseconds>

The interval probe manager will filter probe events by process ID if it is provided in the second field. Assigning the * to the second field indicates that the probe will be fired for all processes. Further, the only value supported by the interval probe manager for the third field is the clock keyword that identifies the probe specification as being for a wall clock probe. The fourth or last field, that is the <# milliseconds> field, identifies the number of milliseconds between firings of the probe. The interval probe manager requires that the value for this field consist only of digits 0-9. For interval probes without process Id, intervals should be exactly divisible by 100. Thus, probe events that are apart by 100ms, 200ms, 300ms, and so on, are allowed in non-profiling interval probes. For interval probes with process Id specified, intervals should be greater or equal to minimum interval allowed for global root user or exactly divisible by 10 for other users. Thus, probe events that are apart by 10ms, 20ms, 30ms, and so on, are allowed for normal users in profiling interval probes. Only one profiling interval probe can be active for a process.

The interval probe manager requires only basic dynamic tracing privileges. The interval probe manager enforces the following limits on the number of probes it supports to prevent malicious users from running the kernel out of memory by creating huge numbers of interval probes.

The interval probe manager does not support the following functions. If used inside an interval manager probe point, these functions will generate an empty string or zero as output.

• get_function
• get_probe
• get_location_point

When process ID is not specified, an interval probe can trigger in the context of any process depending upon when the probe fires since the probe event is based on wall clock time. Because of this, the ProbeVue framework does not allow the use of any of the following functions inside the interval probe manager's action block to prevent unauthorized access to a process's internal data. This security violation is caught only in the kernel. The Vue script will successfully compile but the session will fail to initialize.

• stktrace
• get_userstring

These functions provide no value when used from the probe manager. Even if you are the root user, you cannot call these functions inside the interval probe manager.

When the process ID is specified, the interval probe is triggered for all the threads within the process at the specified time interval. As the probe is fired in the context of the process, stktrace() function and __pname built-in is allowed inside the interval probe manager’s action block, unlike when process ID is not specified.

System trace probe manager

The system trace probe manager provides probe points wherever existing system trace hooks to trace channel zero (system event channel) occur, both within the kernel and within applications. To use this probe manager, you must have the kernel access privilege, and not be running in a WPAR.

The system trace probe manager accepts a 3-tuple probe specification in the following format:

@@systrace:*:<hookid>

where the hookid argument specifies the ID for the specific system trace hook of interest. The hookid argument consists of 4 hex digits typically of the form hhh0. For example, to specify the hookid argument for the fork system call, specify 1390. See the /usr/include/sys/trchkid.h file for examples, such as HKWD_SYSC_FORK. The entries in this file are hook words, where the hookid value is in the upper halfword. Because hook words can be arbitrary, no validation of the hookid argument beyond checking that it is a valid hex string of up to 4 hex digits is performed. It is not an error to specify a hookid value that never occurs.

As a convenience, you can specify the hookid argument with fewer than 4 hex digits. In this case, first a trailing zero is assumed, and then additional leading zeroes as necessary to implicitly define the required 4 digits. For example, you can use 139 as an abbreviation of 1390. Similarly, 0100, 010, and 10 all specify the same hookid value, taken from HKWD_USER1.

You can specify the hookid argument with the * wildcard character. This will probe all system tracing, with likely unacceptable performance implications. Hence, such a specification must be used only when absolutely necessary.

The second tuple is reserved, and must be specified as an asterisk, as shown.

Only system trace events that actually occur and record system trace data trigger probes. In particular, a system trace probe can only occur when system trace is active. The systrace probe manager is an event-based probe manager. Hence, probe name, function name, and location point are not available. As the hookword is passed to the script, this is not a significant restriction.

A non-root user is limited to at most 64 systrace probes simultaneously enabled. No more than 128 explicit systrace probes can be enabled system-wide.

ProbeVue built-in register variables allow access to the data traced. You cannot use the __arg* variables for this purpose. There are two general styles for system tracing.

The following style is for the trchook(64)/utrchook(64) (or the equivalent TRCHKLx macros in C) hooks:

• __r3 contains the 16 bit hookid.
• __r4 contains the subhookid.
• __r5 contains traced data word D1.
• __r6 contains traced data word D2.
• __r7 contains traced data word D3.
• __r8 contains traced data word D4.
• __r9 contains traced data word D5.

Not all trace hooks contain all 5 data words. Undefined data words from a given trace hook will appear as zero. The Vue clause for a given hook ID must know exactly what and how much data its hook ID traces.

If the trace record was produced by one of the functions in the trcgen or trcgent family, use the following style:

• __r3 contains the 16 bit hookid.
• __r4 contains the subhookid.
• __r5 contains traced data word D1.
• __r6 contains the length of the traced data.
• __r7 contains the address of the traced data.

The following script shows a simple example of the systrace probe manager:

	@@systrace:*:1390
	{
		if (__r4 == 0) {	/* normal fork is traced with subhookid zero */
			printf(“HKWD_SYSC_FORK: %d forks child %d\n”, __pid, __r5);
			exit();
		}
	}

System trace must be active for the systrace probe to be triggered.

With appropriate privilege, a Vue script can itself generate system trace records using the "RAS events" Vue functions. However, the systrace probe manager does not detect trace records produced through a Vue script.

Extended system call probe manager (syscallx)

The syscallx probe manager, on the other hand, allows all base system calls to be traced. Base system calls is the set of system calls exported by the kernel and base kernel extensions, which are available immediately after boot-up. System calls that are exported from kernel extensions that may loaded later are not supported. Either a specific system call or all system calls can be specified through the probe point tuple. However, unlike the syscall probe manager, the third field of the probe point tuple for the syscallx must identify the actual kernel entry point function. The syscallx probe manager also limit probes to fire in a specific process if the process ID is specified as the second field of the probe point tuple.

The following are some examples:

/* Probe point tuple to probe the read system call entry for all processes */
@@syscallx:*:kread:entry

/* Probe point tuple to probe the fork system call exit for process with ID 434 */
@@syscallx:434:kfork:exit

/* Probe point tuple to probe entry for all base system calls */
@@syscallx:*:*:entry

/* Probe point tuple to probe exit for all base system calls for process 744 */
@@syscallx:744:*:exit

System calls supported by the syscall probe manager

Running in a WPAR

Workload partitions or WPARs are virtualized operating system environments within a single instance of the AIX operating system. The WPAR environment is somewhat different from the standard AIX operating system environment.

Dynamic tracing is supported in the WPAR environment. By default, when creating a WPAR, only the PV_PROBEVUE_TRC_USER_SELF and the PV_PROBEVUE_TRC_USER privileges are assigned to the WPAR and the superuser (root) on a WPAR system will be granted these privileges. An admin user from the global partition can change the value of the default WPAR privilege set or can explicitly assign additional privileges when creating the WPAR.

Privileges on WPAR have generally the same meanings as on a global partition. Be careful when assigning PV_PROBEVUE_TRC_KERNEL or the PV_PROBEVUE_TRC_MANAGE to a WPAR. Any user with PV_PROBEVUE_TRC_KERNEL privilege can access global kernel variables while a user with PV_PROBEVUE_TRC_MANAGE privilege can change the values of ProbeVue parameters or shutdown ProbeVue. These changes affect all users even those in other partitions.

When you issue the probevue command in a WPAR, processes running in other WPARs or in the global partition are not visible to it. Because of this, you can only probe processes in your same WPAR. The probevue command will fail if the probe specification contains a process ID that is outside its partition. The PV_PROBEVUE_TRC_USER and PV_PROBEVUE_TRC_SYSCALL privileges in a WPAR only allow you to probe user space functions or system calls of processes that are in your WPAR. When probing system calls, the second field of the syscall probe specification must be set to a valid WPAR-visible process ID. Assigning the value * to the second field is not supported.

When a ProbeVue session is initiated in a mobile WPAR, it temporarily switches the WPAR to a non-checkpointable state. After the ProbeVue session terminates, the WPAR is checkpointable again.

@@io:sub_type:io_event:operation_type:filter[|filter …]

@@io:disk:entry:*:hdisk0|FC

long long min_time, max_time;
@@BEGIN {
        min_time = max_time = 0;
}
@@io:disk:entry:*:hdisk0 {
        ts_entry[__iobuf->bufid] = (long long)timestamp();
}
@@io:disk:exit:*:hdisk0 {
        if (ts_entry[__iobuf->bufid]) { /* only if we recorded entry time */
                ts_now = timestamp();
                op_type = (__iobuf->bflags & B_READ) ? "READ" : "WRITE";
                dt = (long long)diff_time(ts_entry[__iobuf->bufid], ts_now, MICROSECONDS);
                if (min_time == 0 || dt < min_time) {
                        min_time = dt;
                        min_blknum = __iobuf->blknum;
                        min_bcount = __iobuf->bcount;
                        min_ts = ts_now;
                        min_optype = op_type;
                }
                if (max_time == 0 || dt > max_time) {
                        max_time = dt;
                        max_blknum = __iobuf->blknum;
                        max_bcount = __iobuf->bcount;
                        max_ts = ts_now;
                        max_optype = op_type;
                }
                ts_entry[__iobuf->bufid] = 0;
        }
}
@@END {
        printf("Maximum and minimum IO operation time for [hdisk0]:\n");
        printf("Max: %lld usec, block=%lld, byte count=%lld, operation=%s, time of operation=[%A]\n",
                max_time, max_blknum, max_bcount, max_optype, max_ts);
        printf("Min: %lld usec, block=%lld, byte count=%lld, operation=%s, time of operation=[%A]\n",
                min_time, min_blknum, min_bcount, min_optype, min_ts);
}

Let this script be in a VUE file named disk_min_max_time.e. It can be executed as:
# probevue disk_min_max_time.e
Let there be some IO activity on hdisk0 (dd command can be used). 
Then after a few minutes, if the above command is terminated (by pressing CTRL-C), then it will print output similar to:
^CMaximum and minimum IO operation time for [hdisk0]:
Max: 48174 usec, block=6927976, byte count=4096, operation=READ, time of operation=[Mar/04/15 03:31:07]
Min: 133 usec, block=6843288, byte count=4096, operation=READ, time of operation=[Mar/04/15 03:31:03]

VUE script to access the raw packet data when the “*” is specified as the protocol.

 /* Define the ether header structure */
struct  ether_header {
        char  ether_dhost[6];
        char  ether_shost[6];
        short ether_type;
};

/* ProbeVue script to access and interpret the data from RAW packet */

@@net:bpf:en0:*:"port 23"
{
        /* define the script local variables */
        __auto struct ether_header eth;
        __auto char *mb;

        /* __mdata contains the address of packet data */
        mb =(char *) __mdata;
        printf("Network probevue\n");
        
       /* 
        * Use already available “copy_kdata(…)” VUE function to copy data of 
        * requested size (size of ether_header) from mbuf data pointer to eth
        * (ether_header) variable.
        */
        copy_kdata (mb, eth);
        printf("Ether Type  from raw data :%x\n",eth.ether_type);
      
}

@@net:tcp:state_change

@@net:udp:sock_recv_buf_overflow

@@net:udp:sock_recv_buf_overflow

IPPROTO_HOPOPTS,
IPPROTO_ICMP, 
IPPROTO_IGMP,
IPPROTO_TCP, 
IPPROTO_UDP, 
IPPROTO_ROUTING,
IPPROTO_FRAGMENT, 
IPPROTO_NONE, 
IPPROTO_LOCAL

ICMP_ECHOREPLY, 
ICMP_UNREACH
ICMP_SOURCEQUENCH,
ICMP_REDIRECT,
ICMP_ECHO, 
ICMP_TIMXCEED,
ICMP_PARAMPROB,
ICMP_TSTAMP,
ICMP_TSTAMPREPLY,
ICMP_IREQ, 
ICMP_IREQREPLY,
ICMP_MASKREQ,
ICMP_MASKREPLY

ICMP_REDIRECT_NET 
ICMP_REDIRECT_HOST 
ICMP_REDIRECT_TOSNET 
ICMP_REDIRECT_TOSHOST

ICMP_TIMXCEED_INTRANS
ICMP_TIMXCEED_REASS

ICMP6_DST_UNREACH
ICMP6_PACKET_TOO_BIG
ICMP6_TIME_EXCEEDED
ICMP6_PARAM_PROB
ICMP6_INFOMSG_MASK
ICMP6_ECHO_REQUEST
ICMP6_ECHO_REPLY 

ICMP6_DST_UNREACH_NOROUTE
ICMP6_DST_UNREACH_ADMIN
ICMP6_DST_UNREACH_ADDR
ICMP6_DST_UNREACH_BEYONDSCOPE
ICMP6_DST_UNREACH_NOPORT

IGMP_HOST_MEMBERSHIP_QUERY
IGMP_HOST_MEMBERSHIP_REPORT
IGMP_DVMRP
IGMP_HOST_NEW_MEMBERSHIP_REPORT
IGMP_HOST_LEAVE_MESSAGE
IGMP_HOST_V3_MEMBERSHIP_REPORT
IGMP_MTRACE
IGMP_MTRACE_RESP
IGMP_MAX_HOST_REPORT_DELAY

DVMPP_PROBE           1
DVMRP_REPORT          2
DVMRP_ASK_NEIGHBORS   3
DVMRP_ASK_NEIGHBORS2  4
DVMRP_NEIGHBORS       5
DVMRP_NEIGHBORS2      6
DVMRP_PRUNE           7
DVMRP_GRAFT           8
DVMRP_GRAFT_ACK       9
DVMRP_INFO_REQUEST    10
DVMRP_INFO_REPLY      11

ARPHRD_ETHER, 
ARPHRD_802_5, 
ARPHRD_802_3, and 
ARPHRD_FDDI

SNAP_TYPE_IP, 
SNAP_TYPE_AP, 
SNAP_TYPE_ARP,
VLAN_TAG_TYPE

ARPOP_REQUEST,
ARPOP_REPLY 

@@net:bpf:en0:tcp:"port 23"
{
        printf("src_addr:%I and dst_addr:%I\n",__ip4hdr->src_addr,__ip4hdr->dst_addr);
        printf("src port:%d\n",__tcphdr->src_port);
        printf("dst port:%d\n",__tcphdr->dst_port);
        printf("tcp hdr_len:%d\n",__tcphdr->hdr_len);
}

Output:
# probevue bpf_tcp.e
src_addr:10.10.10.12 and dst_addr:10.10.18.231
src port:48401
dst port:23
tcp hdr_len:20
..................
.................

@@net:tcp:state_change
when(__proto_info->local_addr ==”10.10.10.1” and __proto_info->remote_addr == 10.10.10.2”
	and __proto_info->local_port =”8000” and  __proto_info->remote_port =”9000”)
{
	printf(“Previous state:%d and current_state:%d\n”,__prev_state,__cur_state);
}

@@net:tcp:listen_q_full
{
	printf(“Listener IP address:%I and Port number is:%d\n”,__proto_info->local_addr, __proto_info->local_port);
}

@@net:udp:sock_recv_buf_overflow
{
        printf("Connection information which drops packet due to socket buffer overflows:\n"); 
        printf("Local IP address:%I and Remote IP address:%I\n",__proto_info->local_addr,__proto_info->remote_addr);
        printf("local port :%d and remote port:%d\n",__proto_info->local_port, __proto_info->remote_port);
 }

@@net:tcp:retransmit
when (__proto_info->local_addr == "10.10.10.1" &&
 	__proto_info->remote_addr == "10.10.10.2" && 
__proto_info->local_port == "4000" &&
 	__proto_info->remote_port == "5000")
{
        printf(" %d th re-transmition for this connection\n", _nth_retransmit);
}

@@net:tcp:send_buf_full
{
        printf("Connection information whenever send buffer full event occurs:\n"); 
        printf("Local IP address:%I and Remote IP address:%I\n",__proto_info->local_addr,__proto_info->remote_addr);
        printf("local port :%d and remote port:%d\n",__proto_info->local_port, __proto_info->remote_port);
 }

__forkfailinfo
{ 
fail_reason; 
}

@@BEGIN 
{ 
         x = 0; 
} 

@@sysproc:forkfail:* 
        when (__forkfailinfo->fail_reason == FAILED_RLIMIT) 
{ 
                printf ("process %s with pid %llu failed to fork a child\n",__pname,__pid); 
                x++; 
} 
 
@@END 
{ 

        printf ("Found %d failures during this vue session\n",x); 
} 

__exitinfo{
	signo;	      	
	returnval;	    
	iscore;	     		
}

echo '@@sysproc:exit:* { printf (" %s    %llu    %llu\n", __pname, __pid,__exitinfo->returnval);}' | probevue

Which will produce an output similar to the following.

 ksh    5833042    0 
 telnetd    7405958    1 
 dumpctrl    7405960    0 
 setmaps    7275006    0 
 termdef    7274752    0 
 hostname    7274754    0 
 id    8257976    0 
 id    8257978    0 
 uname    8257980    0 
 expr    8257982    1 

__threadcreateinfo
{
	tid;	
	pri;       
	policy; 
}

echo '@@sysproc:threadcreate:* 
{ printf ("%s %llu %llu %A\n",__pname,__pid,__threadcreateinfo->tid,timestamp());}' | probevue

nfssync_kproc 5439964 23921151 Feb/22/15 09:22:38 
nfssync_kproc 5439964 24052201 Feb/22/15 09:22:38 
nfssync_kproc 5439964 23920897 Feb/22/15 09:22:38 
nfssync_kproc 5439964 22479285 Feb/22/15 09:22:55 
nfssync_kproc 5439964 23920899 Feb/22/15 09:22:55 
nfssync_kproc 5439964 22479287 Feb/22/15 09:22:55

# echo '@@sysproc:threadterminate:* { printf ("%s %llu %llu %A\n",__pname,__pid,__tid,timestamp());}' | probevue 

A output similar to one shown below can be observed.
nfssync_kproc 5439964 23855555 Feb/22/15 09:59:30 
nfssync_kproc 5439964 21758249 Feb/22/15 09:59:30 
nfssync_kproc 5439964 23855557 Feb/22/15 09:59:30

__threadexceptinfo
{
	pid;          	       
	tid;	       	       
	exception;	      	
	excpt_address        
}

# cat threadexcept.e 
@@sysproc:threadexcept:* 
{ 
 printf ("PID = %llu TID= %llu EXCEPTION=%llu ADDRESS = %llu\n ",__threadexceptinfo->pid,__threadexceptinfo->tid,__threadexceptinfo-
>exception,__threadexceptinfo->excpt_address); 
} 

Run a debugging session on a program compiled with debugging support

# dbx a.out 
Type 'help' for help. 
Core file "core" is older than current program (ignored) 
reading symbolic information ... 
(dbx) stop in main 
[1] stop in main 
(dbx) r 
[1] stopped in main at line 5 
    5           int a=5;

A output similar to one shown below can be observed.

PID = 6816134 TID= 24052015 EXCEPTION=131 ADDRESS = 268436372

__dispatchinfo{
	cpuid;		<- cpu id
	
	oldpid;            <- pid of the thread currently running
	oldtid;		<- thread id of the thread currently running
	oldpriority;	<- priority of the thread currenly running
	newpid;	<- pid of the new process process selected for running 
	newtid;	<- thread id of the thread selected for running
	newpriority;	<-priority of the thread selected for running 
}

_sigsendinfo{
	tpid;               ← target pid
	spid;	 	← source pid  
	signo;	       ← signal sent
}

To continuously print signal source signal target and signal number of all signals.


echo '@@sysproc:sendsig:* {printf ("Source=%llu Target=%llu sig=%llu\n",__sigsendinfo->spid,__sigsendinfo->tpid,__sigsendinfo->signo);}' | 
probevue
 
A output similar to one shown below can be observed.

Source=0 Target=6619618 sig=14 
Source=0 Target=8257944 sig=20 
Source=0 Target=8257944 sig=20

_sigsendinfo{
	tpid;               ← target pid
	spid;	 	← source pid  preprocess.cp
	signo;	       ← signal sent
}

echo '@@sysproc:sigqueue:*{printf ("%llu   %llu   %llu\n",__sigsendinfo->spid,__sigsendinfo->tpid,__sigsendinfo->signo);}' | probevue

A output similar to one shown below can be observed.

8258004   6095294   31

sigdispose

Syntax : @@sysproc:sigdispose:<pid/tid/*>

__sigdisposeinfo{
	tpid;         ← target pid
	ttid;          ← target tid 
	signo;      ← signal whose action is being taken.
	fatal;        ← will be set if the process is going to be killed as part of signal action
} 

cat sigdispose.e

@@sysproc:sigdispose:* 
{ 
 printf ("%llu  %llu %llu %llu\n",__sigdisposeinfo->tpid,__sigdisposeinfo->ttid, __sigdisposeinfo->signo,__sigdisposeinfo->fatal); 
} 

An output similar to one shown below is observed.

5964064  20840935 14 0 
1  65539 14 0 
4719084  19530213 14 0

__sigactioninfo{
	old_sighandle;            ← old signal handler function address
	new_sighandle;	←new signal handler function address 
	signo;		            ← Signal number 
	rpid;		            ← requester's pid
} 

@@sysproc:sighandlestart:* 
{ 
		 
		signal[__tid] = __sighandlestartinfo->signo; 
		printf ("Signal handler at address 0x%x invoked for thread id %llu to handle signal %llu\n",__sighandlestartinfo-
>sighandle,__curthread->tid,__sighandlestartinfo->signo); 
} 


@@sysproc:sighandlefinish:* 
{ 

		printf ("Signal handler completed for thread id %llu for  signal %llu\n",__curthread->tid,signal[__tid]); 
		delete (signal,__tid); 
}

An output similar to the one shown below can be observed.

Signal handler at address 0x20001d58 invoked for thread id 19923365 to handle signal 20 
Signal handler completed for thread id 19923365 for  signal 20 
Signal handler at address 0x10003400 invoked for thread id 20840935 to handle signal 14 
Signal handler completed for thread id 20840935 for  signal 14 
Signal handler at address 0x10002930 invoked for thread id 19530213 to handle signal 14 
Signal handler completed for thread id 19530213 for  signal 14 
Signal handler at address 0x300275d8 invoked for thread id 22348227 to handle signal 14 
Signal handler completed for thread id 22348227 for  signal 14 
Signal handler at address 0x20001a3c invoked for thread id 65539 to handle signal 14 
Signal handler completed for thread id 65539 for  signal 14

__chpriorityinfo{
	pid;
	old_priority;   <- current priority 
	new_priority; <-  new scheduling priority of the thread.
}

echo '@@sysproc:changepriority:* { printf ("%s priority changing from %llu to %llu\n",__pname,__chpriorityinfo->old_priority,__chpriorityinfo-
>new_priority);}' | probevue

An output similar to one shown below can be observed.

xmgc priority changing from 60 to 17 
xmgc priority changing from 17 to 60 
xmgc priority changing from 60 to 17 
xmgc priority changing from 17 to 60 
xmgc priority changing from 60 to 17 

Special built-ins supported

__readyprocinfo{
	pid;		<- process id of thread becoming ready
	tid;		<- Thread id.
	priority;	<- priority of the thread
}

@@BEGIN 
{ 
	printf ("           Pid      Tid      Time           Delta\n"); 
}   

@@sysproc:offreadyq :*
{ 
	ready[__tid] = timestamp(); 
	printf ("offreadyq: %llu %llu %W\n",__readyprocinfo->pid,__readyprocinfo->tid,ready[__tid]); 
} 

@@sysproc:onreadyq :*
{ 
	 
	if (diff_time(ready[__tid],0,MICROSECONDS)) 
	{ 
		auto:diff = diff_time (ready[__tid],timestamp(),MICROSECONDS); 
		printf ("onreadyq : %llu %llu %W      %llu\n",__readyprocinfo->pid,__readyprocinfo->tid,ready[__tid],diff); 
		delete (ready,__tid); 
	} 
}

An output like the one showb below may be observed.

           Pid      Tid      Time           Delta 
offreadyq: 7799280 20709717 5s 679697µs 
onreadyq : 7799280 20709717 5s 679697µs      6 
offreadyq: 7799280 20709717 5s 908716µs 
onreadyq : 7799280 20709717 5s 908716µs      3 
offreadyq: 7799280 20709717 6s 680186µs 
onreadyq : 7799280 20709717 6s 680186µs      5 
offreadyq: 7799280 20709717 6s 710720µs 
onreadyq : 7799280 20709717 6s 710720µs      4 
offreadyq: 7799280 20709717 6s 800720µs 
onreadyq : 7799280 20709717 6s 800720µs      2 
offreadyq: 7799280 20709717 6s 882231µs 
onreadyq : 7799280 20709717 6s 882231µs      2 
offreadyq: 7799280 20709717 6s 962313µs 
onreadyq : 7799280 20709717 6s 962313µs      2 
offreadyq: 7799280 20709717 6s 980311µs 
onreadyq : 7799280 20709717 6s 980311µs      2 

__readyprocinfo{
	pid;		<- process id of thread becoming ready
	tid;		<- Thread id.
	priority;	<- priority of the thread
}

__dispatchinfo{
	cpuid;		<- CPU where selected  thread will run. 
	oldpid;            <- pid of the thread currently running
	oldtid;		<- thread id of the thread currently running
	oldpriority;	<- priority of the thread currenly running
	newpid;	<- pid of the new process process selected for running 
	newtid;	<- thread id of the thread selected for running
	newpriority;	<-priority of the thread selected for running 
}

print process thread id of old and selected thread on CPU '0' with  dispatch time relative to start of the script


echo '@@sysproc:dispatch:* when (__cpuid == 0){printf ("%llu %llu %W\n",__dispatchinfo->oldtid,__dispatchinfo->newtid,timestamp());}' | 
probevue

An output similar to the one shown below can be observed.

24641983 20709717 0s 48126µs 
20709717 23593357 0s 48164µs 
23593357 20709717 0s 48185µs 
20709717 23593357 0s 48214µs 
23593357 20709717 0s 48230µs 
20709717 23593357 0s 48288µs 
23593357 261 0s 48303µs 
261 20709717 0s 48399µs

Example II

Time spent on  CPU '0' by threads in between dispatch event.

@@BEGIN 
{ 
	printf ("Thread cpu Time-Spent\n"); 
} 

@@sysproc:dispatch:* when (__cpuid == $1) 
{ 
	if (savetime[__cpuid] != 0) 
		auto:diff = diff_time (savetime[__cpuid],timestamp(),MICROSECONDS); 
	else 
		diff = 0; 
	savetime[__cpuid] = timestamp(); 
	printf ("%llu %llu %llu\n",__dispatchinfo->oldtid,__dispatchinfo->cpuid,diff); 
}		 
	
# probevue cputime.e 6 
Thread cpu Time-Spent 
3146085 6 0 
3146085 6 9995 
3146085 6 10002 
3146085 6 10008 
3146085 6 99988 
3146085 6 100006 
3146085 6 99995 
3146085 6 99989 
3146085 6 100010 
3146085 6 100001 
3146085 6 100000 
3146085 6 99998 

As can be observed thread 3146085 is being re-dispatched on the CPU at an interval of 1sec in absence of any other thread competing for this 
CPU.

__dispatchinfo{
	cpuid;		<- CPU where selected  thread will run. 
	newpid;	<- pid of the new process process selected for running 
	newtid;	<- thread id of the thread selected for running
	newpriority;	<-priority of the thread selected for running 
}

To print time spent by threads of sysncd on all CPU's
#!/usr/bin/probevue

@@BEGIN

{

	printf ("PROCESSID THREADID CPU TIME\n");

}



@@sysproc:oncpu:$1 

{

	savetime[__cpuid] = timestamp();

}



@@sysproc:offcpu:$1  

{

	if (savetime[__cpuid] != 0)

		auto:diff = diff_time (savetime[__cpuid],timestamp(),MICROSECONDS);

	else

		diff = 0;

	printf ("%llu %llu %llu %llu\n",

		__dispatchinfo->oldpid,

		__dispatchinfo->oldtid,

		__dispatchinfo->cpuid,

		diff);

}


# cputime.e `ps aux|grep syncd| grep -v grep| cut -f 6 -d " "`

An output like on the shown below can be observed.

3735998 18612541 0 2

3735998 15663427 0 1

3735998 15073557 0 1

3735998 18743617 0 1

3735998 18874693 0 1

3735998 18809155 0 15

3735998 18940231 0 20

3735998 18547003 0 1

3735998 19267921 0 1

3735998 19071307 0 17

3735998 18678079 0 1

3735998 18481465 0 1

3735998 19202383 0 15

3735998 19005769 0 1

3735998 19136845 0 19

3735998 6160689 0 190

__dispatchinfo{
	cpuid;		<- CPU where selected  thread will run. 
	newpid;	<- pid of the new process process selected for running 
	newtid;	<- thread id of the thread selected for running
	newpriority;	<-priority of the thread selected for running 
}

__sleepinfo{
	pid;
	tid;
	waitchan;   <-- wait channel of this sleep.	
}

__foldcpuinfo{
	cpuid;		<- logical cpu id which triggers core folding 
	gpcores;         <- general purpose (unfolded, non-exclusive) cores available.
}

__foldcpuinfo{
	cpuid;		<- logical cpu id which triggers core folding 
	gpcores;         <- general purpose (unfolded, non-exclusive) cores available.
}

__bindprocessorinfo{
	ispid       <- 1 if cpu is bound to process; 0 for a thread 
	id;	    <- thread or process id.
	
	cpuid;
	
};

__changecpuinfo
{
	oldcpuid;	<-source CPU
	newcpuid; 	<- target CPU   
	pid;
	tid;		<-Thread id
}

@@sysproc:changecpu:*

{
printf ("changecpu PID=%llu TID=%llu old_cpuid=%d new_cpuid= %d \n",
__changecpuinfo->pid,__changecpuinfo->tid,__changecpuinfo->oldcpuid,__changecpuinfo->newcpuid);

}


An output like the one shown below may be observed.

changecpu PID=852254 TID=1769787 old_cpuid=26 new_cpuid= 27 

changecpu PID=852254 TID=1769787 old_cpuid=-1 new_cpuid= 0 

changecpu PID=852254 TID=1769787 old_cpuid=0 new_cpuid= 1 

changecpu PID=852254 TID=1769787 old_cpuid=1 new_cpuid= 2 

__srcresourceinfo{
	type;
	subtype;
	id;		<- resource type identifier
	offset; 		<-offset if a memory resource
	length;		<- length if a memory resource
	policy;
}
__tgtresourceinfo{
	type;
	subtype;
	id;		<- resource type identifier
	offset;		<-offset if a memory resource
	length;		<- length if a memory resource
	policy;		
}

__srcresourceinfo{
	type;
	subtype;
	id;		<- resource type identifier
	offset; 		<-offset if a memory resource
	length;		<- length if a memory resource
	policy;
}

__tgtresourceinfo{
	type;
	subtype;
	id;		<- resource type identifier
	offset;		<-offset if a memory resource
	length;		<- length if a memory resource
	policy;		
}

__drphaseinfo{
	dr_operation;   ← dr operation
	dr_flags;	
	dr_phase;
	handler_rc;    ← always 0 in drphasestart
}