Topic
7 replies Latest Post - ‏2012-11-21T09:50:22Z by mwandishi
starndawn
starndawn
37 Posts
ACCEPTED ANSWER

Pinned topic failed to open db2diag.log

‏2012-11-20T07:38:29Z |
OS: redhat 5.3 64bit
DB2: "DB2 v8.1.3.156", "s080813", "MI00287", and FixPak "17".

system message shows:
Nov 18 20:17:40 prddb2 DB211734: Open of log file "/db2home/db2prod1/sqllib/db2dump/db2diag.log" failed with rc 0x860F0004
Nov 18 20:17:46 prddb2 DB211734: Open of log file "/db2home/db2prod1/sqllib/db2dump/db2diag.log" failed with rc 0x860F0004
Nov 18 20:17:46 prddb2 DB21528: Open of log file "/db2home/db2prod1/sqllib/db2dump/db2diag.log" failed with rc 0x860F0004
Nov 18 20:17:46 prddb2 DB21523: Open of log file "/db2home/db2prod1/sqllib/db2dump/db2diag.log" failed with rc 0x860F0004
Nov 18 20:17:46 prddb2 DB21528: Open of log file "/db2home/db2prod1/sqllib/db2dump/db2diag.log" failed with rc 0x860F0004

file permission:
-rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log

Looks like the file hasn't been updated since 2 years ago, but I can open it with VI. Thoughts? Thank you very much!
Updated on 2012-11-21T09:50:22Z at 2012-11-21T09:50:22Z by mwandishi
  • mwandishi
    mwandishi
    47 Posts
    ACCEPTED ANSWER

    Re: failed to open db2diag.log

    ‏2012-11-20T08:39:37Z  in response to starndawn
    Hi,

    Firstly, the RC ( 0x860F0004 ) means disk error:

    db2diag -rc 0x860F0004 | egrep -i -A 4 "Description|Associated information"
    Description:
    general failure (DD)

    Associated information:
    Sqlcode -980
    SQL0980C A disk error occurred. Subsequent SQL statements cannot be processed.

    The fact that it is 0 bytes is odd too? What I can't see from the ls of the file is the path it is in. The messages file output shows it is looking in: /db2home/db2prod1/sqllib/db2dump but is it really in there now? I am wondering if this is an old log file in an old path

    My questions are:

    1) What is DIAGPATH set too ( db2 get dbm cfg| grep -i diagp ) ?
    2) Is db2diag.log actually in: /db2home/db2prod1/sqllib/db2dump ?
    3) What path does the db2diag.log you showed us actually reside in ?
    4) When you open it in vi is it empty ?
    5) It is unlikely but it may have a file lock on it, what does fuser or lsof report against this file ?
    6) Why is this in the messages file, what monitoring is in place to check this file and is it checking the right thing in the right place ?
    7 ) In v8 there is a tool called db2diag, if you run it with no flags it will pick up and read the default db2diag.log, does this return any data, for example: db2diag -t 2012-11-10 ( will return diag data from the 10th of November onwards. )
    8) Once you know the real path ( point 1) ) can you cd to that path and run:
    8.1) pwd
    8.2) ls -la db2diag* to show any / all db2diag.logs, including archived ones
    9) I assume the instance is started and running OK ?
    10) Assuming this: -rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log' is the current file it is owned by db2prod1, perhaps the thing that is monitoring it and driving the message has a different UID and doesn't have access?

    Best regards,
    Stephen.
    • starndawn
      starndawn
      37 Posts
      ACCEPTED ANSWER

      Re: failed to open db2diag.log

      ‏2012-11-20T09:04:15Z  in response to mwandishi
      Hi mwandishi,

      Thank you for your reply!

      1) What is DIAGPATH set too ( db2 get dbm cfg| grep -i diagp ) ?
      I have double checked this, and the db2diag.log is in this directory.
      Diagnostic data directory path (DIAGPATH) = /db2home/db2prod1/sqllib/db2dump

      2) Is db2diag.log actually in: /db2home/db2prod1/sqllib/db2dump ?
      yes.
      db2prod1@prddb2:/db2home/db2prod1ls -l /db2home/db2prod1/sqllib/db2dump
      total 1314848
      -rw-r--r-- 1 db2prod1 db2prdi 1552462 Mar 16 2011 a19913.000
      -rw-r--r-- 1 db2prod1 db2prdi 1562858 Mar 16 2011 a21628.000
      -rw-r--r-- 1 db2prod1 db2prdi 676488 Mar 16 2011 a21643.000
      -rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log
      -rw-r----- 1 db2prod1 db2prdi 6291376 Nov 5 04:26 db2eventlog.000

      3) What path does the db2diag.log you showed us actually reside in ?
      /db2home/db2prod1/sqllib/db2dump

      4) When you open it in vi is it empty ?
      yes, it's empty, I can even copy this file, and remove the copy.

      5) It is unlikely but it may have a file lock on it, what does fuser or lsof report against this file ?
      it returns nothing after issuing these 2 commands on this file

      6) Why is this in the messages file, what monitoring is in place to check this file and is it checking the right thing in the right place ?
      not sure why, maybe db2 cannot write messages into this file, then reports to the system.

      7 ) In v8 there is a tool called db2diag, if you run it with no flags it will pick up and read the default db2diag.log, does this return any data, for example: db2diag -t 2012-11-10 ( will return diag data from the 10th of November onwards. )
      the command returns nothing, I tried the 3 recent days for it.

      8) Once you know the real path ( point 1) ) can you cd to that path and run:
      8.1) pwd
      8.2) ls -la db2diag* to show any / all db2diag.logs, including archived ones
      I also tried to find the hidden files, but nothing ...
      db2prod1@prddb2:db2dump$ pwd
      /db2home/db2prod1/sqllib/db2dump
      db2prod1@prddb2:db2dump$ ls -la db2diag*
      -rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log

      9) I assume the instance is started and running OK ?
      yep, it's running fine.

      10) Assuming this: -rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log' is the current file it is owned by db2prod1, perhaps the thing that is monitoring it and driving the message has a different UID and doesn't have access?
      the db2 instance owner is db2prod1, so I think this is not the case.
      • mwandishi
        mwandishi
        47 Posts
        ACCEPTED ANSWER

        Re: failed to open db2diag.log

        ‏2012-11-20T09:10:03Z  in response to starndawn
        Hi,

        No problem. One question I forgot to ask, what is the DIAGLEVEL?

        Stephen
        • starndawn
          starndawn
          37 Posts
          ACCEPTED ANSWER

          Re: failed to open db2diag.log

          ‏2012-11-20T09:17:22Z  in response to mwandishi
          Diagnostic error capture level (DIAGLEVEL) = 3
          • mwandishi
            mwandishi
            47 Posts
            ACCEPTED ANSWER

            Re: failed to open db2diag.log

            ‏2012-11-20T11:13:25Z  in response to starndawn
            So we know that:

            db2diag.log is in the correct place, the permissions & ownerships are correct, DIAGLEVEL is default and db2 is up meaning we should be getting some messages. However, it is 0 bytes and we have a 'disk error'.

            I assume there is sufficient disk space in the path where it resides ?

            Can you touch a new file in there ( touch test.log in /db2home/db2prod1/sqllib/db2dump as the instance owner ) what happens ?

            Also, I wonder if there is a delayed value for DIAGLEVEL / DIAGPATH, can you try this:

            db2 attach to db2prod1
            db2 get dbm cfg show detail | egrep -i "diaglevel|diagpath"

            Also what happens if you do this:

            db2diag -A
            db2 update dbm cfg using diaglevel 4
            Wait a while...
            check to see if anything is added.

            What happens if you move it, albeit temporarily:

            db2 update dbm cfg using diagpath /.../.../
            ( where is is a different path that db2prod1 has rw access too )

            If we still don't get a db2diag.log then I'd used strace of the db2sycs PID to see what problems it is reporting in accessing the file.

            Stephen.
            • starndawn
              starndawn
              37 Posts
              ACCEPTED ANSWER

              Re: failed to open db2diag.log

              ‏2012-11-21T03:24:32Z  in response to mwandishi
              Hi Stephen,

              Thank you for your suggestion!

              The log file cannot be written, it reported "Operation not permitted". We used lsattr/chattr to change the hidden file permission of this log file, and removed it successfully. Now it is ok for db2 instance to write messages into this file.
              • mwandishi
                mwandishi
                47 Posts
                ACCEPTED ANSWER

                Re: failed to open db2diag.log

                ‏2012-11-21T09:50:22Z  in response to starndawn
                Hi,

                No problem and you are welcome. Glad it is working normally again.

                Stephen.