== PROBEVUE - MONITOR OUTPUT FILE SIZE ==

Body

When using 'probevue' the output file can grow very quickly.
This can become an issue with disk space. If the file becomes too
big one has to interrupt the 'probevue' script, remove or archive
the file and then restart the script. This is not efficient and we
might miss important events while the script is not running.

So far 'probevue' can't be used with the shell '|' and this
can make controlling the output file size a bit tricky.

Imagine you have a program that fails because a call to 'fstat()'
fails with 'EBADF'. You would want to monitor the process for
'open/close/fstat' system calls so that you can get the name of
the file and check if that file is not 'opened' at the time 'fstat()'
fails. If the program is something like a database server it might
run for days on and the output file would quickly become too big
to be stored on disk. What would be noce however would be to keep
only a given amount of the 'latest' traces produced by 'probevue'.

This article describes how to do that using a 'named pipe' and a C
program to monitor the output of the 'probevue' program and keep
only a given amount of data.

We have a 'probevue' script that will write to a 'named pipe'. On the
other side of that 'named pipe' will be a C program that will check
write the data received to a file. When the file reaches a size provided
by the user the C program will archive the file. Each time a file is
archived the program checks that only the last 2 files (configurable)
are kept. The oldest file will be deleted.

Example:

  [fstat1.pb]
  We will have a 'probevue' script to monitor the 'open/close/fstat' system
  calls for a given process of which the user will provide the pid.

  [fstat1_collect.sh]
  A shell script will create a 'named pipe' and start the 'probevue'
  script directing it to write to that 'named pipe'.

  [fstat1_monitor.c]
  A C program will also be starting on the 'read' side of the
  'named pipe' to collect the data sent by 'probevue'. Once the file
  reaches a given size it will be archived. And the C program will only
  keep the latest 2 files of data.

Files are at the end of this article but let's first see how this would work.
In this case we will run the script against a 'db2' database server:

  [Get the pid of the process to monitor]

  # db2pd -edu | grep "^db2sysc"
  db2sysc PID: 43712894

  [Start the script to collect data]

  # fstat1.collect.sh 43712894
  Start at 2018-09-13 01:25:20

  create named pipe                 : passed
  start fstat1_monitor              : passed
  start probevue script             : passed
  You can leave both probevue and fstat1_monitor running until the problem
  happens again. When the problem happens please do the following:

    db2pd -edu > db2pd.edu

  Then kill the following processes:

   52691250 pts/44  0:00 dd bs=4k of=./fstat1pipe
   36962902 pts/44  0:00 dd bs=4k of=./fstat1pipe
   30539956 pts/44  0:00 probevue -t 10 -s 64 fstat1.pb 43712894
   55116426 pts/44  0:00 ./fstat1_monitor ./fstat1pipe fstat1.out 1048576

  IMPORTANT: If the db2 engine is restarted the programs should be
  killed and that script started again.

  End at 2018-09-13 01:25:20

After a while you will see files of size 'MAXSIZE' (defined in 'fstat1.collect.sh')
being created with only the latest 2 files available:

  # ls -l fstat*
  -rwxr-xr-x    1 dalla    build          2457 Sep 13 01:24 fstat1.collect.sh
  -rw-r--r--    1 root     system            0 Sep 13 01:25 fstat1.collect.sh.out
  -rwxr--r--    1 root     system      1048695 Sep 13 01:41 fstat1.out.2
  -rwxr--r--    1 root     system      1050522 Sep 13 01:43 fstat1.out.3
  -rwxr--r--    1 root     system        71450 Sep 13 01:43 fstat1.out.4
  -rwxr-xr-x    1 dalla    build          1900 Sep 13 01:14 fstat1.pb
  -rwxr-xr-x    1 dalla    build         11452 Sep 13 01:14 fstat1_monitor
  -rw-r--r--    1 root     system            0 Sep 13 01:25 fstat1_monitor.out
  prw-r--r--    1 root     system            0 Sep 13 01:43 fstat1pipe

As you can see only 2 'complete' files of data are kept plus the current one.
Once you have the data you need you can kill the process that are not needed
anymore (as mentioned above):

  # kill 30539956 52691250 36962902 55116426 

The output files will contain the data you need:

  [db2sysc - 43712894 - 20447571] close(29)
  [db2sysc - 43712894 - 20447571] open(/home3/dalla/sqllib/db2systm) = 28 [errno 0]
  [db2sysc - 43712894 - 20447571] close(28)
  [db2sysc - 43712894 - 20447571] close(27)
  [db2sysc - 43712894 - 20447571] close(26)
  [db2sysc - 43712894 - 20447571] open(/home3/dalla/dalla/NODE0000/SQL00001
                                       /MEMBER0000/SQLDBCONF) = 26 [errno 0]
  [db2sysc - 43712894 - 20447571] open(/home3/dalla/dalla/NODE0000/SQL00001
                                       /SQLOGCTL.GLFH.1) = 27 [errno 0]
  [db2sysc - 43712894 - 20447571] open(/home3/dalla/dalla/NODE0000/SQL00001
                                       /SQLOGCTL.GLFH.2) = 28 [errno 0]


============================
== FILE fstat1.collect.sh ==
============================

#! /bin/ksh

MAXSIZE=1048576
RPOUTFILE=fstat1_monitor.out
PVOUTFILE=probevue.out
TARGETPID=$1
PIPENAME=./fstat1pipe
PROBEFILE=fstat1.out



#------------------------------------------------------------------------------#
#
# Cleanup.
#
#------------------------------------------------------------------------------#
rm -f $PROBEFILE* $PIPENAME $RPOUTFILE $PVOUTFILE

DT=`date +"%Y-%m-%d %H:%M:%S"`
echo "Start at $DT"
echo ""

if test "X$TARGETPID" = "X"
then
    echo "Usage: $0 <db2sysc pid>"
    exit 1
fi


#------------------------------------------------------------------------------#
#
# Create a pipe for transfering data.
#
#------------------------------------------------------------------------------#
echo "create named pipe                 : \c"
mkfifo $PIPENAME 1>>$0.out 2>&1
if test $? -eq 0
then
    echo "passed"
else
    echo "failed"
    exit 1
fi


#------------------------------------------------------------------------------#
#
# Start fstat1_monitor program first.
#
#------------------------------------------------------------------------------#
echo "start fstat1_monitor              : \c"

nohup ./fstat1_monitor $PIPENAME $PROBEFILE $MAXSIZE 1>>$RPOUTFILE 2>&1 &
echo "passed"



#------------------------------------------------------------------------------#
#
# Start probevue script. Script is started in the background and the
# output is piped to 'dd' to write to a fifo. 'probevue' cannot
# output directly to a fifo.
#
#------------------------------------------------------------------------------#
echo "start probevue script             : \c"

nohup probevue -t 10 -s 64 fstat1.pb $TARGETPID | dd bs=4k of=$PIPENAME 1>>$PVOUTFILE    2>&1 &
echo "passed"



#------------------------------------------------------------------------------#
#
# We are done.
#
#------------------------------------------------------------------------------#
echo "You can leave both probevue and fstat1_monitor running until the problem"
echo "happens again. When the problem happens please do the following:"
echo ""
echo "  db2pd -edu > db2pd.edu"
echo ""
echo "Then kill the following processes:"
echo ""
ps | grep dd | grep $PIPENAME | grep -v grep
ps | grep probevue | grep fstat1 | grep -v grep
ps | grep fstat1_monitor | grep -v grep
echo ""
echo "IMPORTANT: If the db2 engine is restarted the programs should be"
echo "killed and that script started again."
echo ""

DT=`date +"%Y-%m-%d %H:%M:%S"`
echo "End at $DT"
echo ""


============================
== FILE fstat1.pb         ==
============================

/*
 * fstat1.pb: Try to catch 'errno 9' when running 'fstat()'.
 *
 * Run as user 'root' using the following command line:
 *
 *     probevue -t 10 -e 75 -s 64 -o fstat1.out fstat1.pb <pid>
 *
 * In case 'probevue' returns an error you can modify the
 * settings accordingly:
 *
 * Example to modify the number of threads that can be traced:
 *
 *     probevctrl -c "num_threads_traced=2048"
 *
 * Example to modify the per cpu buffer size:
 *
 *     probevctrl -c "default_buffer_size=64"
 *
 * Example to modify the maximum amount of memory used by probevue:
 *
 *     probevctrl -c "max_total_mem_size=256"

 *
 *
 * dalla
 */

int                        open(char *);
int                        fstatx(int);
int                        close(int);

__thread int               in_open;
__thread char             *open_pathname;
__thread int               in_fstatx;
__thread int               fstatx_fd;


@@syscall:$1:open:entry
{
    __auto String fname[256];

    fname = get_userstring((void *) __arg1, -1);
    thread:open_pathname = __arg1;
    thread:in_open = 1;
}

@@syscallx:$1:fstatx:entry
{
    thread:fstatx_fd = __arg1;
    thread:in_fstatx = 1;
}

@@syscall:$1:close:entry
{
    printf("[%s - %d - %d] close(%d)\n",
           __pname, __pid, __tid, __arg1);
}


@@syscall:$1:open:exit
when (thread:in_open == 1)
{
    __auto String fname[256];

    fname = get_userstring((void *) thread:open_pathname, -1);
    printf("[%s - %d - %d] open(%s) = %d [errno %d]\n",
         __pname, __pid, __tid, fname, __rv, __errno);
    thread:in_open = 0;
}

@@syscallx:$1:fstatx:exit
when (thread:in_fstatx == 1)
{
    if ((__rv == -1) && (__errno == 9)) {
        printf("[%s - %d - %d] fstatx(%d) = %d [errno = %d]\n",
               __pname, __pid, __tid, thread:fstatx_fd, __rv, __errno);
    }
    thread:in_fstatx = 0;
}


=================================================
== FILE fstat1_monitor.c                       ==
==                                             ==
== cc -q64 fstat1_monitor.c -o fstat1_monitor  ==
=================================================

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <string.h>
#include <signal.h>
#include <unistd.h>

#define BUFSIZE 4096
#define PATHSIZE 128

int                     interrupt;

void                    sig_handler(int, siginfo_t *, void *);


int
main(int ac, char **av)
{
    int                 ifd;
    int                 ofd;
    int                 num;
    int                 perms;
    size_t              maxsize;
    size_t              cursize;
    ssize_t             cc;
    char               *bf;
    char                fname[PATHSIZE];
    struct sigaction    sa;


    /*
     * Open input and output files.
     */
    interrupt = 0;

    if (ac < 4) {
        printf("[error] Usage: %s <input> <output> <filesize>\n", av[0]);
        exit(1);
    }

    if ((ifd = open(av[1], O_RDONLY)) < 0) {
        printf("[error] open(%s), [errno = %d]\n", av[1], errno);
        exit(1);
    }


    /*
     * Install a signal handler for interrupt.
     */
    (void) memset((void *) &sa, 0, sizeof(struct sigaction));
    sa.sa_flags = SA_SIGINFO;
    sa.sa_sigaction = sig_handler;
    sigfillset(&sa.sa_mask);
    if (sigaction(SIGINT, &sa, 0)) {
        printf("[error] sigaction(SIGINT), [errno = %d]\n", errno);
        return -1;
    }

    (void) memset((void *) &sa, 0, sizeof(struct sigaction));
    sa.sa_flags = SA_SIGINFO;
    sa.sa_sigaction = sig_handler;
    sigfillset(&sa.sa_mask);
    if (sigaction(SIGTERM, &sa, 0)) {
        printf("[error] sigaction(SIGTERM), [errno = %d]\n", errno);
        return -1;
    }


    num = 0;
    sprintf(fname, "%s.%d", av[2], num);
    perms = S_IRWXU|S_IRGRP|S_IROTH;
    if ((ofd = open(fname, O_WRONLY|O_TRUNC|O_CREAT, perms)) < 0) {
        printf("[error] open(%s), [errno = %d]\n", fname, errno);
        exit(1);
    }

    if ((bf = malloc(BUFSIZE)) == 0) {
        printf("[error] malloc(%d), [errno = %d]\n", BUFSIZE, errno);
        exit(1);
    }

    maxsize = atol(av[3]);
    cursize = 0;



    /*
     * Keep reading data from the pipe.
     */
    while ((interrupt == 0) && ((cc = read(ifd, bf, BUFSIZE)) > 0)) {
        if (write(ofd, bf, cc) < 0) {
            printf("[error] write(%d), [errno = %d]\n", cc, errno);
            exit(1);
        }

        cursize += cc;

        if (cursize >= maxsize) {
            close(ofd);

            if (num >= 2) {
                sprintf(fname, "%s.%d", av[2], num - 2);
                unlink(fname);
            }

            num++;
            sprintf(fname, "%s.%d", av[2], num);
            if ((ofd = open(fname, O_WRONLY|O_TRUNC|O_CREAT, perms)) < 0) {
                printf("[error] open(%s), [errno = %d]\n", fname, errno);
                exit(1);
            }

            cursize = 0;
        }
    }

    close(ifd);
    close(ofd);


    exit(0);
}

void
sig_handler(int sig, siginfo_t *info, void *ctx)
{
    interrupt++;

    printf("[info] [%d] received signal %d\n", getpid(), sig);

    return;
}