DDTS problem-solving and debugging: tools and techniques

Resolving The Problem

Problem-solving in DDTS is a fairly complicated process which usually requires assistance from Technical Support; however, there are a set of common information sources and problem-solving steps that can sometimes help solve a problem without contacting Support, or can facilitate getting the problem solved most quickly when you do contact Support.

Here, in no particular order, are descriptions of a set of useful information sources and commands you can use to gain insight into DDTS's operation:

BASIC DDTS AND ENVIRONMENTAL INFORMATION

Important Commands to know are:

Command	Description
ddtsversion	Displays the version number of DDTS
uname -a	Displays the OS and release you're running
df -k	Displays filesystem statistics; tells you if the disk is full; DDTS is notoriously sensitive to the filesystem filling up! It also shows whether the filesystem is local to the machine you are on or NFS-mounted from another host (this can be important when dealing with permission problems).
echo $PATH	Shows your UNIX PATH (PATH is used to find commands and programs you type at the UNIX command line, and DDTS also uses it to locate filters (programs) referenced in master.tmpl.
which xddts	Another way of verifying the PATH; this command shows you where in the PATH a particular command (in this case 'xddts') would be found.

The 'ddts' userid on UNIX, and in fact any DDTS user accessing DDTS via the UNIX-hosted facilities, should have their PATH set to include the DDTS '*_bin' directory appropriate for their UNIX architecture; this is *NOT NECESSARILY* ~ddts/bin!

If you suspect a path problem, consult the DDTS Installation & Licensing Guide where it describes setting up the path and using the ddts_archdir.sh program (pages 1-15..1-17 in the release 4.1 guide) or in the DDTS User's Guide (appendix B).

NOTE:

1. In versions of DDTS prior to 3.1, the 'ddtsversion' command was named 'whichddts'. If you get "command not found" when running 'ddtsversion' and your path is set correctly, then you should run 'whichddts' and you should be advised that your release of DDTS is obsolete.
2. Certain UNIX operating systems have variations on the command names or syntax; for example, the 'df' command is called 'bdf' on HP-UX. See your UNIX system administrator, or the UNIX manpages, or a book on your specific UNIX system for more details.
   

---

DDTS's LOG:

DDTS keeps a runtime log of information about its processing. This file is in ~ddts/spool/ and its filename is "LOG" (Note: all uppercase). This file is extremely useful in tracking down problems having to do with connecting DDTS sites, mail notification, failures to commit defect changes, and a variety of other issues.

There is a part of the nightly 'ddtsclean' program which checks the LOG file's size and archives it (moves it to "LOG.old") if it is beyond a certain size. This will happen every day on a very-active system or less often on a system with moderate activity. Therefore in normal operation on a very-active system, the LOG file will contain activity from 1:00am that day, and LOG.old will have information about the previous day. On less-active systems the date range will be wider.

Each line in the LOG file is of the following format:

daemon YYMMDD HHMMSS Some text.....

Where:
daemon is the name of the daemon that produced the LOG message YYMMDD HHMMSS is the date and time the message was written "Some text...." is the specific text of the message

Generally in debugging situations involving the DDTS daemons, it is important to focus on only one daemon at a time; for example, in debugging notification-mail-related issues you would look *only* for lines that had 'bugmail' in the first column.

Here is a list of DDTS's main daemons:

Daemon	Description
ddtsd	Process dispatcher; starts the others when they have work
ddtsserver	The SQL database engine
bugs.in	Commits defect change transactions to SQL and allbugs
ddtsbackend	Runs after each defect change transaction
bugmail	Sends notification mail
ddupd	Handles duplicate processing
mail.in	Reads incoming mail
net.out	Sends mail to connected/subscribed sites

There are a few other daemons as well, but let's not confuse this discussion by listing them all. These are the main ones.

These daemons each have a manpage which you can consult to get more information; for example, type 'man ddtsd' to learn more about ddtsd.

To debug an issue in any of these aspects of DDTS's processing, look on the LOG for the time period around the time the transaction took place, and focus on messages from the daemon that handles that type of work; for example, as stated above, for a problem with email notification you would look for lines that say "bugmail" in the first column, where the
message text contains the defect ID of the defect you're having a problem with.

---

MONITORING THE LOG FILE IN REAL TIME

So far you have learned how to look at the LOG for events in the past; their data is already written in the LOG so you could just pull up the LOG file in an editor, or 'more', or print it, etc.

However, to diagnose situations in real time it is often useful to run the following:

cd ~ddts/spool
tail -f LOG&

The UNIX 'tail' command displays the last few lines of a file. The '-f'
operand on 'tail' keeps the command running and shows you any new lines that get added to the file, in real time; this essentially transforms your terminal window into a scrolling console display of everything that's going on with DDTS at any point in time.

Make sure your window is wide and long enough to read the messages without their scrolling by too fast or line-wrapping excessively.

---

WHAT'S RIGHT AND WRONG IN THE LOG?

It may be difficult, in looking at the LOG, to distinguish normal conditions from abnormal ones. Therefore it is useful to look at the pattern of lines produced by a normal activity, and compare it to the lines produced when a failure occurs; you can often diagnose the situation based on what is *not* there (but was expected to be there).

In this regard, a useful exercise is to run the 'tail -f' command as shown above, and watch your DDTS LOG. If your system is relatively idle, the display may be relatively static as there is no DDTS-related work going on.

Go to a DDTS user interface and commit a defect-change transaction (submit a new defect or modify an existing one, and click Commit, to commit the change). Observe the pattern of lines on the LOG.

You should see the following:

• Several daemons will get invoked in a series, including bugs.in, bugmail, ddupd, ddtsbackend, and possibly others depending on the specifics of the transaction.
• For each daemon, there is a particular pattern to the output: messages that the daemon has started (this message includes the UNIX process ID which is very useful for debugging), progress messages, and finally a termination message.
• The bugs.in daemon is always executed first, but several of the other "down-stream" daemons may execute in an unpredictable or overlapping order; that's OK, because the tasks they carry out are unrelated and therefore it's safe to do them in parallel (for example ddupd and bugmail).

Observing the lines in the LOG under normal circumstances will allow you to discern anomalous situations much more easily.