The first time you sign into developerWorks, a profile is created for you. Select information in your profile (name, country/region, and company) is displayed to the public and will accompany any content you post. You may update your IBM account at any time.

All information submitted is secure.

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerworks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

All information submitted is secure.

My developerWorks:

My notifications:

Sign out

Select a language:

Solutions for tracing UNIX applications

Martin Brown (mc@mcslp.com), Professional writer, Freelance

Martin Brown has been a professional writer for over eight years. He is the author of numerous books and articles across a range of topics. His expertise spans myriad development languages and platforms -- Perl, Python, Java, JavaScript, Basic, Pascal, Modula-2, C, C++, Rebol, Gawk, Shellscript, Windows, Solaris, Linux, BeOS, Mac OS/X and more -- as well as Web programming, systems management and integration. Martin is a regular contributor to ServerWatch.com, LinuxToday.com and IBM developerWorks, and a regular blogger at Computerworld, The Apple Blog and other sites, as well as a Subject Matter Expert (SME) for Microsoft. He can be contacted through his Web site at http://www.mcslp.com.

Summary: If you are developing a UNIX® application, then you can trace and debug the running application and extract the information you need from it. But what if you want to know what is going on inside a UNIX application and you don't have access to the source code?This tutorial looks at some systems that enable you to trace the execution of applications and work out what they are doing without having to make any modifications to the source code, and even without having to stop and restart the application.

Date: 31 Mar 2009
Level: Intermediate
PDF: A4 and Letter (66 KB | 24 pages)Get Adobe® Reader®

Activity: 44199 views
Comments:

Tracing network packets

Tracing the application code can tell you what is going on within the application, and it can show you what a system is doing, but what if you want to monitor a network application and find out what information is being transferred over the network? There are many tools available, but the snoop tool in Solaris, iptrace in AIX and the Ethereal tool for many different platforms all provide the ability to view the packets going by on the network.

Network tracing basics

Nearly all of the different tools for tracing network packets work in the same basic fashion. Normally, your computer reads packets from the network and only processes and acts on the network packets that were specifically sent to your computer. With older Ethernet networks, all packets were sent to all computers. With a network switch, if you want to view packets other than those for your computer, you may need to connect to the management port on your network.

The methods for using these tools and the information they can provide differ slightly, but the fundamentals are the same. With AIX, the iptrace is a background daemon, so you must explicitly start and stop the tool to switch the process on and off. For example, to start, type: # startsrc -s iptrace -a "-i tr0 /home/user/iptrace/log1". To stop the tool, type: # stopsrc -s iptrace.

On Solaris, the snoop tool is an application that you can execute at will: # snoop.

However, the number of packets on a typical network may be too much to be decoded and displayed on the fly. In this case, you can either specify a limiting scan, by setting the port name or number, or the computer name or number, or output the raw data to a file and then post-process the file. You can record the information by setting the output file: $ snoop -o networkdump.log.

To read in the saved data, type: $ snoop -i networkdump.log.

When using snoop for tracing applications, you will probably want to be more specific about your searches.

Scanning for a specific host with snoop

Typical uses for snoop during tracing are to find if an application is communicating over the network, or to find the specifics of what information is being exchanged.

For example, if you are trying to diagnose a problem between an NFS server and the NFS client, you might want to ensure that the two machines are actually exchanging information. One way to do this is to use snoop to monitor the data exchanged. On the client, you could use snoop to log the data being transferred to and from the server.

For example, Listing 17 looks for traffic to the host called bear.

Listing 17. Looking for traffic to host called bear


$ snoop host bear    
Using device rge0 (promiscuous mode)
tweedledee.mcslp.pri -> bear.mcslp.pri TCP D=2049 S=1014 Syn Seq=1160567073 
    Len=0 Win=49640 Options=<mss 1460,nop,wscale 0,nop,nop,sackOK>
bear.mcslp.pri -> tweedledee.mcslp.pri TCP D=1014 S=2049 Syn Ack=1160567074 
    Seq=498630824 Len=0 Win=5840 Options=<mss 1460,nop,nop,sackOK,nop,wscale 7>
tweedledee.mcslp.pri -> bear.mcslp.pri TCP D=2049 S=1014 Ack=498630825 
    Seq=1160567074 Len=0 Win=49640
tweedledee.mcslp.pri -> bear.mcslp.pri NFS C ACCESS3 FH=EC09 (read,lookup,
    modify,extend,delete)
bear.mcslp.pri -> tweedledee.mcslp.pri TCP D=1014 S=2049 Ack=1160567230 
    Seq=498630825 Len=0 Win=54
bear.mcslp.pri -> tweedledee.mcslp.pri NFS R ACCESS3 OK (read,lookup,modify,
    extend,delete)
tweedledee.mcslp.pri -> bear.mcslp.pri TCP D=2049 S=1014 Ack=498630949 
    Seq=1160567230 Len=0 Win=49640
tweedledee.mcslp.pri -> bear.mcslp.pri NFS C GETATTR3 FH=EC09
bear.mcslp.pri -> tweedledee.mcslp.pri NFS R GETATTR3 OK
tweedledee.mcslp.pri -> bear.mcslp.pri TCP D=2049 S=1014 Ack=498631065 
    Seq=1160567382 Len=0 Win=49640
tweedledee.mcslp.pri -> bear.mcslp.pri NFS C GETATTR3 FH=02E4
bear.mcslp.pri -> tweedledee.mcslp.pri NFS R GETATTR3 OK
tweedledee.mcslp.pri -> bear.mcslp.pri NFS C GETATTR3 FH=8F7F
bear.mcslp.pri -> tweedledee.mcslp.pri NFS R GETATTR3 OK
tweedledee.mcslp.pri -> bear.mcslp.pri NFS C GETATTR3 FH=2764
bear.mcslp.pri -> tweedledee.mcslp.pri NFS R GETATTR3 OK
tweedledee.mcslp.pri -> bear.mcslp.pri NFS C GETATTR3 FH=FD7F
bear.mcslp.pri -> tweedledee.mcslp.pri NFS R GETATTR3 OK
tweedledee.mcslp.pri -> bear.mcslp.pri NFS C GETATTR3 FH=FF7F
bear.mcslp.pri -> tweedledee.mcslp.pri NFS R GETATTR3 OK
tweedledee.mcslp.pri -> bear.mcslp.pri NFS C GETATTR3 FH=8F7F
bear.mcslp.pri -> tweedledee.mcslp.pri NFS R GETATTR3 OK
tweedledee.mcslp.pri -> bear.mcslp.pri NFS C GETATTR3 FH=725E
bear.mcslp.pri -> tweedledee.mcslp.pri NFS R GETATTR3 OK
tweedledee.mcslp.pri -> bear.mcslp.pri NFS C GETATTR3 FH=8D7F
bear.mcslp.pri -> tweedledee.mcslp.pri NFS R GETATTR3 OK
tweedledee.mcslp.pri -> bear.mcslp.pri NFS C GETATTR3 FH=0C64
bear.mcslp.pri -> tweedledee.mcslp.pri NFS R GETATTR3 OK
tweedledee.mcslp.pri -> bear.mcslp.pri NFS C GETATTR3 FH=6CEC
bear.mcslp.pri -> tweedledee.mcslp.pri NFS R GETATTR3 OK
tweedledee.mcslp.pri -> bear.mcslp.pri NFS C GETATTR3 FH=5997
bear.mcslp.pri -> tweedledee.mcslp.pri NFS R GETATTR3 OK
tweedledee.mcslp.pri -> bear.mcslp.pri TCP D=2049 S=1014 Ack=498632341 
    Seq=1160569142 Len=0 Win=49640

Fortunately, you can see that data is being exchanged. Snoop is capable of decoding the contents of the raw packets to determine what data is actually being exchanged. In this case, the GETATTR3 network packet with NFS is being used, which gets the attribute data for a file (such as the name, size, and ownership information).

There is a lot of additional information being shown, so you could also be more explicit and select only the NFS packets. You can combine the expression used to specify what data to select by combining them like this: $ snoop host bear and protocol nfs.

To get more detailed information, you can use the verbose mode to output the actual data being exchanged. With the verbose mode enabled, snoop will debug each component of the packet, from the raw Ethernet header data to the individual TCP and protocol information. A single packet showing the output when filesystem state data is requested over NFS is shown in Listing 18.

Listing 18. A single packet showing the output when filesystem state data is requested over NFS


ETHER:  ----- Ether Header -----
ETHER:  
ETHER:  Packet 2 arrived at 16:09:4.99083
ETHER:  Packet size = 142 bytes
ETHER:  Destination = 0:1a:ee:1:1:c0, 
ETHER:  Source      = 0:1d:60:1b:9a:2d, 
ETHER:  Ethertype = 0800 (IP)
ETHER:  
IP:   ----- IP Header -----
IP:   
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:         .... ..0. = not ECN capable transport
IP:         .... ...0 = no ECN congestion experienced
IP:   Total length = 128 bytes
IP:   Identification = 61800
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 64 seconds/hops
IP:   Protocol = 6 (TCP)
IP:   Header checksum = c7b7
IP:   Source address = 192.168.0.2, bear.mcslp.pri
IP:   Destination address = 192.168.0.5, tweedledee.mcslp.pri
IP:   No options
IP:   
TCP:  ----- TCP Header -----
TCP:  
TCP:  Source port = 2049
TCP:  Destination port = 1013 (Sun RPC)
TCP:  Sequence number = 2161119694
TCP:  Acknowledgement number = 1253508400
TCP:  Data offset = 20 bytes
TCP:  Flags = 0x18
TCP:        0... .... = No ECN congestion window reduced
TCP:        .0.. .... = No ECN echo
TCP:        ..0. .... = No urgent pointer
TCP:        ...1 .... = Acknowledgement
TCP:        .... 1... = Push
TCP:        .... .0.. = No reset
TCP:        .... ..0. = No Syn
TCP:        .... ...0 = No Fin
TCP:  Window = 348
TCP:  Checksum = 0xec08
TCP:  Urgent pointer = 0
TCP:  No options
TCP:  
RPC:  ----- SUN RPC Header -----
RPC:  
RPC:  Record Mark: last fragment, length = 84
RPC:  Transaction id = 485864481
RPC:  Type = 1 (Reply)
RPC:  This is a reply to frame 1
RPC:  Status = 0 (Accepted)
RPC:  Verifier   : Flavor = 0 (None), len = 0 bytes
RPC:  Accept status = 0 (Success)
RPC:  
NFS:  ----- Sun NFS -----
NFS:  
NFS:  Proc = 18 (Get filesystem statistics)
NFS:  Status = 0 (OK)
NFS:  Post-operation attributes:  (not available)
NFS:  Total space = 488217268224 bytes
NFS:  Available space = 137675571200 bytes
NFS:  Available space - this user = 112875532288 bytes
NFS:  Total file slots = 60555264
NFS:  Available file slots = 58563011
NFS:  Available file slots - this user = 58563011
NFS:  Invariant time = 0 sec
NFS:

Looking for specific information can also be useful if you want to determine what different applications are doing. For example, you may want to monitor what Web sites are being accessed from a host, which you can do by looking for the HTTP protocol and only outputting the HTTP lines from the snoop output (see Listing 19).

Listing 19. Monitoring what Web sties are being accessed from a host


$ snoop -v port 80 |egrep '^HTTP'
...
HTTP: ----- HyperText Transfer Protocol -----
HTTP: 
HTTP: GET / HTTP/1.0
HTTP: User-Agent: Wget/1.10.2
HTTP: Accept: */*
HTTP: Host: www.bbc.co.uk
HTTP: Connection: Keep-Alive
HTTP: [...]
...

The egrep selects only the lines starting with HTTP. Here we can see somebody has accessed the BBC Web site.

Comments