Adding DTrace probes to your applications

DTrace provides a rich environment of probes that can be used to monitor the execution of your system, from the kernel up to your application. You can perform a significant amount of examination without changing your application, but to get detailed statistics, you need to add probes to your application. In this article we will examine how to design the probes, where to add them into your application, the best locations for the probes, and how to effectively build and use the probes that you have added.

Share:

Martin Brown (mc@mcslp.com), Freelance Writer, Author

Martin Brown has been a professional writer for more than seven years. He is the author of numerous books and articles across a range of topics. His expertise spans myriad development languages and platforms -- Perl, Python, Java™, JavaScript, Basic, Pascal, Modula-2, C, C++, Rebol, Gawk, Shellscript, Windows®, Solaris, Linux, BeOS, Mac OS X and more -- as well as Web programming, systems management, and integration. He is a Subject Matter Expert (SME) for Microsoft® and regular contributor to ServerWatch.com, LinuxToday.com, and IBM developerWorks. He is also a regular blogger at Computerworld, The Apple Blog, and other sites. You can contact him through his Web site.



04 May 2010

Also available in Chinese

Introduction

The Dynamic Tracing (DTrace) functionality built into Solaris (including OpenSolaris), FreeBSD and Mac OS X provides a simple environment for tracing applications dynamically. Unlike debugging, DTrace can be switched on or off at will, and you do not need to provide a special build of your application to take advantage of the tracing functionality.

All the above platforms support the use of standard DTrace probes. Those exposed by the operating system on the boundary of different functions within the code are covered. These probes, known as a Function Boundary Tracing (FBT), allow you to identify when execution of a given function starts or stops.

The limit of this functionality is that it can only be used for probing functions, rather than functional fragments, of an application. If you want to examine the execution across a number of functions that make up a single operation, or if you want to look at a fragment of a single function, the FBT cannot help.

For your own applications, you can address this by using the User-land Statically Defined Tracing (USDT). USDT enables you as a developer to add specific probes to your application at points in the code that you have identified as significant. The USDT system also enables you to expose data from your running application, which can be accessed as arguments to the probes when tracing the application.

Before you start adding USDT probes to your system, you first need to consider what you want the probes to report, what information they might provide, and the potential performance issues.


Probe design

Once you decide that the standard FBT probes are not suitable for your requirements, you need to start thinking about adding static probes to your application.

The first consideration to make when using DTrace to add probes to your application is to determine what you are actually using the probes for in the first place. Probes can help you identify a wide range of issues and information, but you should be targeting your probes. You should pick out specific areas, such as functionality, performance, or other measurable information that goes beyond the scope of what you could find just using the standard entry/exit probes provided on all function barriers.

At a simple level, therefore, you should consider that there are two primary types of probes:

  1. Information probes: These expose or summarize a piece of information that would otherwise be difficult to determine during the execution of a program. Good examples here include the size or contents of internal structures, or the operation or triggering of an event that isn't handled directly by a function. There are many examples of this in the existing operating system probes. For example, you can get information about the disk I/O statistics or faults within the virtual memory system.
  2. Operational probes: These bracket a specific event or series of statements so that you can use them to get specific information about internal structures at the start and end of a sequence, or to monitor the execution time of a specific set of statements. Because you can put these probes anywhere to indicate the start and end of a operation, they can span multiple functions, or only a small fragment of a given function. These are more practical than using the function boundaries which may provide too much, or too little, scope. As a convention these probes usually have a suffix of start and done.

Once you have determined the type of probes, the next consideration is whether you want to expose any additional information in the probes, and if so, what information, and in what format, you want to provide them. Within DTrace, probes can expose information through arguments which are available for processing when writing a suitable DTrace script or one-liner. For example, if you were instrumenting a file I/O function, you might add the name of the file being written to the probe for that function.

When defining the probes, these arguments are given names and the type: probe write__file__start(int id, char *filename);.

When used within a DTrace script during monitoring, each argument is available within variables named arg0, arg1, etc. So you could print the filename, the second argument, using: printf("%s\n", copyinstr(arg1));.

Choosing the right data to expose is a case of understanding what you want to get from the probe while monitoring your application. In the I/O function example above, for example, knowing the filename may be critical if the function is used to write to multiple files. But there is no point in exposing this information in the probe if the function only writes to the same file.

Therefore, you must consider how to present the information. Do you want to be able to summarize data by an operation type, or by the files or network ports? Do you want to know data sizes, or the actual data being written? All of these are very program and environment specific.

There is also an overhead with providing information in this way, and you should try to be sparing with sharing information, particularly large structures. Instead, you should try to provide limited information, or if possible summarized information (although be aware that copying, shortening or reformatting strings for the purposes of DTrace probes will have an even bigger impact).

There are two methods of introducing probes in this way that can be useful: multiple probes, and using the special 'is probe enabled?' bracketing. The latter solution is supported in a very simple fashion when you write the probes and generate a header file using the dtrace command. You can surround blocks of code with a determination of whether the probe has been enabled (for instance, it is actively being monitored), and perform additional operations, which can be useful if you need to summarize or massage the data (see Listing 1).

Listing 1. Code block to see if the probe has been enabled
if (WRITE_FILE_START_ENABLED())
{
    ...
}

The former method, using separate probes, enables users to use the specific probes they want to get the information they need. For example, using our I/O example, you might have a probe structure that provides different information at different levels, with the probes notionally nested so that you can determine the information you want:

  • write-file-start (id)
  • write-file-data(filename, buffer)
  • write-file-done (id)

When monitoring the application, if all you want is to monitor the speed of the write-file operation, you can use the write-file-start and write-file-done probes, which provide an ID for referencing. If you want the filename data, you could optionally monitor the write-file-data probe to output that information during processing.

Finally, with all probes it is worth remembering that anybody with rights to monitor DTrace probes has access to the information that you expose. If you expose e-mail addresses or message contents, for example, in your probes, then any user with DTrace rights will be able to read the content. Be careful about exposing potentially sensitive information in this way. If you can, either provide only the statistical data, or, if you must expose real information that might be sensitive, you could consider obscuring the data so that the true content can not be determined.


Defining the probes

On Solaris/OpenSolaris you can define probes using the macros in /usr/include/sys/sdt.h. The macros enable you to insert probes within your code by calling the appropriate macro for the arguments that you want to include. For example, to insert a probe with no arguments you can use: DTRACE_PROBE("prime","calc-start").

If you want to share arguments, there is a different macro, numbered one to five for sharing the corresponding number of arguments when the probe is triggered. For example, to share one argument: DTRACE_PROBE("prime","calc-start",prime).

This method is only supported on Solaris/OpenSolaris. For a more portable version supported on Solaris/OpenSolaris, FreeBSD and Mac OS X (and one that also provides an easier method for inserting probes into your code), you can create a probe definition file that will contain each of the probes that you want to insert into your code, including the definition of the arguments that you want to share through each probe.

The format of the file is similar to C. You must specify one or more providers, and within each provider, specify each probe that you want to support in your code. You can see a sample of a probe definition in code Listing 2.

Listing 2. Sample probe definition
provider primes {

/* Start of the prime calculation */

   probe primecalc__start(long prime);

/* End of the prime calculation */

   probe primecalc__done(long prime, int isprime);

/* Exposes the size of the table of existing primes */

   probe primecalc__tablesize(long tablesize);

};

The provider is the name of the provider once the probes are installed within an application. Probe names within DTrace are identified by the provider, module, function and probe name: provider:module:function:name. For USDT probes, you can specify only the provider and name portions of this specification.

The name of the probe is taken from the string immediately after the probe keyword. You can separate words within the probe name using a double underscore. This is converted to a single hyphen when you want to use the probe during tracing. For example, the primecalc__start() probe name in this file can be identified using the primes::primecalc-start name (combining the provider and probe name).

The arguments to each probe in the definition are used to help identify the data types for the arguments within the C code. Remember that each argument is available during tracing as arg0, arg1, argN, and so on. So in the case of the primecalc__done probe, the prime number would be arg0 and whether the number is a prime would be arg1.

Once you have a probe definition file, you use the dtrace command to convert the probe definitions into a header file: $ dtrace -o probes.h -h -s probes.d.

The above command specifies the name of the output file (-o), to generate a header (-h) and the name of the source probe definitions file (-s).

The resulting header file contains macros that you can place into your code to insert the probes. You can use these as many times as you like wherever you want the probes to trigger within the code.

The format of the probe macros follows the probe names you defined. For example, the primecalc__done macro is PRIMES_PRIMECALC_DONE. Now that you have a probe definition file (which will also be used when building the application) and the header file, it is time to insert the probes into your C source.


Identifying locations for probes

For the purposes of describing where to put the probes, we're going to look at a simple program for determining prime numbers. The code is hardly the most efficient method, but the DTrace probes may help us to identify the problems.

The original source code can be seen in Listing 3.

Listing 3. Source code for program for determining prime numbers
#include <stdio.h>

long primes[1000000] = { 3 };
long primecount = 1;

int main(int argc, char **argv)
{
  long divisor = 0;
  long currentprime = 5;
  long isprime = 1;

  while (currentprime < 1000000)
    {
      isprime = 1;
       for(divisor=0;divisor<primecount;divisor++)
        {
          if (currentprime % primes[divisor] == 0)
            {
              isprime = 0;
            }
        }
      if (isprime)
        {
          primes[primecount++] = currentprime;
          printf("%d is a prime\n",currentprime);
        }
      currentprime = currentprime + 2;
    }
}

The modified code, with the DTrace probes added to it, is shown in Listing 4.

Listing 4. Modified code with DTrace probes added
#include <stdio.h>
#include "probes.h"

long primes[1000000] = { 3 };
long primecount = 1;

int main(int argc, char **argv)
{
  long divisor = 0;
  long currentprime = 5;
  long isprime = 1;

  while (currentprime < 1000000)
    {
      isprime = 1;
      PRIMES_PRIMECALC_START(currentprime);
      for(divisor=0;divisor<primecount;divisor++)
        {
          if (currentprime % primes[divisor] == 0)
            {
              isprime = 0;
            }
        }
      PRIMES_PRIMECALC_DONE(currentprime,isprime);
      if (isprime)
        {
          primes[primecount++] = currentprime;
          PRIMES_PRIMECALC_TABLESIZE(primecount);
          printf("%d is a prime\n",currentprime);
        }
      currentprime = currentprime + 2;
    }

}

The location of the probes took into account a number of different factors:

  • The header file generated by dtrace has been included into the source code.
  • For the primecalc-start and primecalc-done probes, notice how the probes have been placed immediately outside the main loop that performs the calculation. The temptation is to put the probes maybe at the start and end of the loop, because this seems like a logical place to include the probes. But as mentioned earlier, for probes like these where you want to monitor a specific area of functionality, you should be as close to the real operation that you want to monitor. If the probes had been placed at the start and end of the while loop, a number of operations unrelated to the actual calculation of the primes would have been included. Although it is unlikely to make a significant difference in this application, in others, such additional steps could have added significant time to the operation that you really want to monitor.
  • The primecalc-tablesize probe is not designed to be monitored in terms of the time it takes, but you do want to monitor the size of the table effectively. The most obvious place to put this is nearest to any change in the value. This is important because from a tracking perspective you will want to know exactly where the value changed, even if you are not monitoring the changes over time.
  • Note that the done probe provides both the number and whether the number was determined as being prime. You actually know whether the number is prime or not, immediately after the end of the for loop, even though you don't use the determined value until the if statement. Also, note that by providing the value of the isprime variable, you can put the done probe into the code once with that value as an argument, in place of using different probes. When it comes to scripting, you can use the value to count the time taken for primes and non-primes by using predicates.
  • For other applications, the same rule applies. You want to ensure that any probe providing statistical (but not necessarily timing) data is as close to any changes in that information as possible. Remember that you can place the same probe into the code multiple times. In this case, you could have placed the PRIMES_PRIMECALC_TABLESIZE macro at the head of the main code block immediately after the variable initialization so that the initial value could be highlighted.

Compiling the application

You can compile a DTrace application in much the same as any other, using your chosen C compiler. On Mac OS X/FreeBSD, you can simply compile your application as normal. But on Solaris/OpenSolaris you must modify the object file and generate a new object file containing the DTrace probes.

On Solaris/OpenSolaris, the process requires modifying your object files before final linking, and you must link in the separately generated object file that contains the probes you want to enable in your application. The process of modifying the object files occurs in place — that is, you specify the object file and the process modifies the file saving the changes back into the source file. The object file is generated as part of this process. The general sequence is:

  1. Compile each source file to an object file, for example: $ gcc -c primes.c.
  2. Once you have compiled all of the source files, create a DTrace probe object file that contains the probes to be linked into the main program. For example, for a single file you can do this using the following command: $ dtrace -G -s probes.d -o probes.o primes.o
  3. The above reads the object file primes.o and the probe definition probes.d and then generates a probe file called probes.o. If you have multiple object files with DTrace probes, you can specify any additional object files on the command line. For example: $ dtrace -G -s probes.d -o probes.o file1.o file2.o file3.o.
  4. Link your application, including all of the object files and the generated probe object file: $ gcc -o primes primes.o probes.o.
  5. The final primes executable will be probed, enabled, and ready to use.

On FreeBSD/Mac OS X, you do not have to generate a separate probe object file for linking. This makes the compilation process much more straightforward:

  1. Compile each source file to an object file, for example: $ gcc -c primes.c.
  2. Link your application, including all of the object files: $ gcc -o primes primes.o.
  3. Your application with DTrace probes is ready for use.

Now let's try using the probes.


Writing scripts to use your probes

You have only added some very basic probes to what is a very basic program, but you can still get some useful information out of the execution. For example, you can use the start and done probes to find out how long it is taking to find all of the prime numbers and all of the non-prime numbers. Since the distribution of primes over non-primes is much smaller, you should see a significant difference between the timing of these two elements. A sample script to show this is shown in Listing 5.

Listing 5. Script showing difference in finding prime numbers
#!/usr/sbin/dtrace -s

#pragma D option quiet

primes*:::primecalc-start
{
  self->start = timestamp;
}

primes*:::primecalc-done
/arg1 == 1/
{
        @times["prime"] = sum(timestamp - self->start);
}

primes*:::primecalc-done
/arg1 == 0/
{
        @times["nonprime"] = sum(timestamp - self->start);
}

END
{
        normalize(@times,1000000);
        printa(@times);
}

This script aggregates the timing data from the start probe to the done probe, using a predicate to identify whether the calculation was ultimately for a prime or non-prime number. The information is put into an associative array, aggregated using the sum() aggregate function.

The normalize() function in the END block divides the result by a million to get the timing in milliseconds. If you try running this script at the same time as the primes program is running, you will get output similar to that in Listing 6.

Listing 6. Output
$ dtrace -s timing.d            
^C

  prime                                             10784
  nonprime                                       16340221

The results show an expected higher time spent looking for non-primes rather than primes. This is because there are comparatively many more non-prime numbers than primes.


Gotchas and traps

There are many issues to consider when including DTrace probes, but here are some known issues that you may want to be aware of:

  • Avoid placing probes at function entry and exit points. Since you can already access these automatically using the FBT probes, the only benefit to this process is providing alternative names for the probes than the function names. If you are adding USDT probes, you should be creating probes according to the operational points you want to probe, not the function boundaries.
  • Avoid putting DTrace probes as the last statement within a function. On some platforms and compiler combinations, optimization of the code may result in the probe either being optimized away (effectively removing the probe entirely), or the probe could get attached to the calling function (which may remove the argument data, or lead to interesting timing issues).
  • If you are compiling a library for linking and want to make the probes available, then you need to run the DTrace process on the object files before you create the library. In addition, on Solaris/OpenSolaris, you will need to include the generated object file in the library along with the original object files. Additional care needs to be taken when using complex build processes such as those applied by automake, which can put an object files into a temporary directory to be used explicitly during library generation. You must make sure that you are running the dtrace command on the object file that gets added to the library.
  • The probe object file you generate, and the object files on which it was based, must match. If the object files change, the probe object file must be regenerated or the linking will fail.
  • If you are using autoconf, cmake, or similar, and want to retain cross-platform compatibility, the only consideration for dtrace is the usage of -G to generate the probe object file. You can easily provide a test for this during configuration.

Summary

Using the function-based tracing with DTrace already provides you with a lot of information, but the greatest flexibility comes from adding your own static probes. With a static probe you can choose its location, name, and the information that it exposes, enabling you to fine tune the information that you expose to match the data that you want to extract.

Adding static probes is a case of creating a suitable definition file, and then using the macros generated from this file within your C source code. Although there are some minor steps to complete depending on your platform, compilation is comparatively straightforward. Providing you keep in mind the advice about where to put your probes and how to choose the probes you want to use, the benefits will usually outweigh the small overhead of actually including the probes within your application.

Resources

Learn

Get products and technologies

  • Innovate your next open source development project with IBM trial software, available for download or on DVD.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=486910
ArticleTitle=Adding DTrace probes to your applications
publish-date=05042010