Topic
  • 7 replies
  • Latest Post - ‏2012-11-21T09:50:22Z by mwandishi
starndawn
starndawn
37 Posts

Pinned topic failed to open db2diag.log

‏2012-11-20T07:38:29Z |
OS: redhat 5.3 64bit
DB2: "DB2 v8.1.3.156", "s080813", "MI00287", and FixPak "17".

system message shows:
Nov 18 20:17:40 prddb2 DB211734: Open of log file "/db2home/db2prod1/sqllib/db2dump/db2diag.log" failed with rc 0x860F0004
Nov 18 20:17:46 prddb2 DB211734: Open of log file "/db2home/db2prod1/sqllib/db2dump/db2diag.log" failed with rc 0x860F0004
Nov 18 20:17:46 prddb2 DB21528: Open of log file "/db2home/db2prod1/sqllib/db2dump/db2diag.log" failed with rc 0x860F0004
Nov 18 20:17:46 prddb2 DB21523: Open of log file "/db2home/db2prod1/sqllib/db2dump/db2diag.log" failed with rc 0x860F0004
Nov 18 20:17:46 prddb2 DB21528: Open of log file "/db2home/db2prod1/sqllib/db2dump/db2diag.log" failed with rc 0x860F0004

file permission:
-rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log

Looks like the file hasn't been updated since 2 years ago, but I can open it with VI. Thoughts? Thank you very much!
Updated on 2012-11-21T09:50:22Z at 2012-11-21T09:50:22Z by mwandishi
  • mwandishi
    mwandishi
    47 Posts

    Re: failed to open db2diag.log

    ‏2012-11-20T08:39:37Z  
    Hi,

    Firstly, the RC ( 0x860F0004 ) means disk error:

    db2diag -rc 0x860F0004 | egrep -i -A 4 "Description|Associated information"
    Description:
    general failure (DD)

    Associated information:
    Sqlcode -980
    SQL0980C A disk error occurred. Subsequent SQL statements cannot be processed.

    The fact that it is 0 bytes is odd too? What I can't see from the ls of the file is the path it is in. The messages file output shows it is looking in: /db2home/db2prod1/sqllib/db2dump but is it really in there now? I am wondering if this is an old log file in an old path

    My questions are:

    1) What is DIAGPATH set too ( db2 get dbm cfg| grep -i diagp ) ?
    2) Is db2diag.log actually in: /db2home/db2prod1/sqllib/db2dump ?
    3) What path does the db2diag.log you showed us actually reside in ?
    4) When you open it in vi is it empty ?
    5) It is unlikely but it may have a file lock on it, what does fuser or lsof report against this file ?
    6) Why is this in the messages file, what monitoring is in place to check this file and is it checking the right thing in the right place ?
    7 ) In v8 there is a tool called db2diag, if you run it with no flags it will pick up and read the default db2diag.log, does this return any data, for example: db2diag -t 2012-11-10 ( will return diag data from the 10th of November onwards. )
    8) Once you know the real path ( point 1) ) can you cd to that path and run:
    8.1) pwd
    8.2) ls -la db2diag* to show any / all db2diag.logs, including archived ones
    9) I assume the instance is started and running OK ?
    10) Assuming this: -rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log' is the current file it is owned by db2prod1, perhaps the thing that is monitoring it and driving the message has a different UID and doesn't have access?

    Best regards,
    Stephen.
  • starndawn
    starndawn
    37 Posts

    Re: failed to open db2diag.log

    ‏2012-11-20T09:04:15Z  
    • mwandishi
    • ‏2012-11-20T08:39:37Z
    Hi,

    Firstly, the RC ( 0x860F0004 ) means disk error:

    db2diag -rc 0x860F0004 | egrep -i -A 4 "Description|Associated information"
    Description:
    general failure (DD)

    Associated information:
    Sqlcode -980
    SQL0980C A disk error occurred. Subsequent SQL statements cannot be processed.

    The fact that it is 0 bytes is odd too? What I can't see from the ls of the file is the path it is in. The messages file output shows it is looking in: /db2home/db2prod1/sqllib/db2dump but is it really in there now? I am wondering if this is an old log file in an old path

    My questions are:

    1) What is DIAGPATH set too ( db2 get dbm cfg| grep -i diagp ) ?
    2) Is db2diag.log actually in: /db2home/db2prod1/sqllib/db2dump ?
    3) What path does the db2diag.log you showed us actually reside in ?
    4) When you open it in vi is it empty ?
    5) It is unlikely but it may have a file lock on it, what does fuser or lsof report against this file ?
    6) Why is this in the messages file, what monitoring is in place to check this file and is it checking the right thing in the right place ?
    7 ) In v8 there is a tool called db2diag, if you run it with no flags it will pick up and read the default db2diag.log, does this return any data, for example: db2diag -t 2012-11-10 ( will return diag data from the 10th of November onwards. )
    8) Once you know the real path ( point 1) ) can you cd to that path and run:
    8.1) pwd
    8.2) ls -la db2diag* to show any / all db2diag.logs, including archived ones
    9) I assume the instance is started and running OK ?
    10) Assuming this: -rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log' is the current file it is owned by db2prod1, perhaps the thing that is monitoring it and driving the message has a different UID and doesn't have access?

    Best regards,
    Stephen.
    Hi mwandishi,

    Thank you for your reply!

    1) What is DIAGPATH set too ( db2 get dbm cfg| grep -i diagp ) ?
    I have double checked this, and the db2diag.log is in this directory.
    Diagnostic data directory path (DIAGPATH) = /db2home/db2prod1/sqllib/db2dump

    2) Is db2diag.log actually in: /db2home/db2prod1/sqllib/db2dump ?
    yes.
    db2prod1@prddb2:/db2home/db2prod1ls -l /db2home/db2prod1/sqllib/db2dump
    total 1314848
    -rw-r--r-- 1 db2prod1 db2prdi 1552462 Mar 16 2011 a19913.000
    -rw-r--r-- 1 db2prod1 db2prdi 1562858 Mar 16 2011 a21628.000
    -rw-r--r-- 1 db2prod1 db2prdi 676488 Mar 16 2011 a21643.000
    -rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log
    -rw-r----- 1 db2prod1 db2prdi 6291376 Nov 5 04:26 db2eventlog.000

    3) What path does the db2diag.log you showed us actually reside in ?
    /db2home/db2prod1/sqllib/db2dump

    4) When you open it in vi is it empty ?
    yes, it's empty, I can even copy this file, and remove the copy.

    5) It is unlikely but it may have a file lock on it, what does fuser or lsof report against this file ?
    it returns nothing after issuing these 2 commands on this file

    6) Why is this in the messages file, what monitoring is in place to check this file and is it checking the right thing in the right place ?
    not sure why, maybe db2 cannot write messages into this file, then reports to the system.

    7 ) In v8 there is a tool called db2diag, if you run it with no flags it will pick up and read the default db2diag.log, does this return any data, for example: db2diag -t 2012-11-10 ( will return diag data from the 10th of November onwards. )
    the command returns nothing, I tried the 3 recent days for it.

    8) Once you know the real path ( point 1) ) can you cd to that path and run:
    8.1) pwd
    8.2) ls -la db2diag* to show any / all db2diag.logs, including archived ones
    I also tried to find the hidden files, but nothing ...
    db2prod1@prddb2:db2dump$ pwd
    /db2home/db2prod1/sqllib/db2dump
    db2prod1@prddb2:db2dump$ ls -la db2diag*
    -rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log

    9) I assume the instance is started and running OK ?
    yep, it's running fine.

    10) Assuming this: -rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log' is the current file it is owned by db2prod1, perhaps the thing that is monitoring it and driving the message has a different UID and doesn't have access?
    the db2 instance owner is db2prod1, so I think this is not the case.
  • mwandishi
    mwandishi
    47 Posts

    Re: failed to open db2diag.log

    ‏2012-11-20T09:10:03Z  
    • starndawn
    • ‏2012-11-20T09:04:15Z
    Hi mwandishi,

    Thank you for your reply!

    1) What is DIAGPATH set too ( db2 get dbm cfg| grep -i diagp ) ?
    I have double checked this, and the db2diag.log is in this directory.
    Diagnostic data directory path (DIAGPATH) = /db2home/db2prod1/sqllib/db2dump

    2) Is db2diag.log actually in: /db2home/db2prod1/sqllib/db2dump ?
    yes.
    db2prod1@prddb2:/db2home/db2prod1ls -l /db2home/db2prod1/sqllib/db2dump
    total 1314848
    -rw-r--r-- 1 db2prod1 db2prdi 1552462 Mar 16 2011 a19913.000
    -rw-r--r-- 1 db2prod1 db2prdi 1562858 Mar 16 2011 a21628.000
    -rw-r--r-- 1 db2prod1 db2prdi 676488 Mar 16 2011 a21643.000
    -rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log
    -rw-r----- 1 db2prod1 db2prdi 6291376 Nov 5 04:26 db2eventlog.000

    3) What path does the db2diag.log you showed us actually reside in ?
    /db2home/db2prod1/sqllib/db2dump

    4) When you open it in vi is it empty ?
    yes, it's empty, I can even copy this file, and remove the copy.

    5) It is unlikely but it may have a file lock on it, what does fuser or lsof report against this file ?
    it returns nothing after issuing these 2 commands on this file

    6) Why is this in the messages file, what monitoring is in place to check this file and is it checking the right thing in the right place ?
    not sure why, maybe db2 cannot write messages into this file, then reports to the system.

    7 ) In v8 there is a tool called db2diag, if you run it with no flags it will pick up and read the default db2diag.log, does this return any data, for example: db2diag -t 2012-11-10 ( will return diag data from the 10th of November onwards. )
    the command returns nothing, I tried the 3 recent days for it.

    8) Once you know the real path ( point 1) ) can you cd to that path and run:
    8.1) pwd
    8.2) ls -la db2diag* to show any / all db2diag.logs, including archived ones
    I also tried to find the hidden files, but nothing ...
    db2prod1@prddb2:db2dump$ pwd
    /db2home/db2prod1/sqllib/db2dump
    db2prod1@prddb2:db2dump$ ls -la db2diag*
    -rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log

    9) I assume the instance is started and running OK ?
    yep, it's running fine.

    10) Assuming this: -rw-rw-rw- 1 db2prod1 db2prdi 0 Dec 9 2010 db2diag.log' is the current file it is owned by db2prod1, perhaps the thing that is monitoring it and driving the message has a different UID and doesn't have access?
    the db2 instance owner is db2prod1, so I think this is not the case.
    Hi,

    No problem. One question I forgot to ask, what is the DIAGLEVEL?

    Stephen
  • starndawn
    starndawn
    37 Posts

    Re: failed to open db2diag.log

    ‏2012-11-20T09:17:22Z  
    • mwandishi
    • ‏2012-11-20T09:10:03Z
    Hi,

    No problem. One question I forgot to ask, what is the DIAGLEVEL?

    Stephen
    Diagnostic error capture level (DIAGLEVEL) = 3
  • mwandishi
    mwandishi
    47 Posts

    Re: failed to open db2diag.log

    ‏2012-11-20T11:13:25Z  
    • starndawn
    • ‏2012-11-20T09:17:22Z
    Diagnostic error capture level (DIAGLEVEL) = 3
    So we know that:

    db2diag.log is in the correct place, the permissions & ownerships are correct, DIAGLEVEL is default and db2 is up meaning we should be getting some messages. However, it is 0 bytes and we have a 'disk error'.

    I assume there is sufficient disk space in the path where it resides ?

    Can you touch a new file in there ( touch test.log in /db2home/db2prod1/sqllib/db2dump as the instance owner ) what happens ?

    Also, I wonder if there is a delayed value for DIAGLEVEL / DIAGPATH, can you try this:

    db2 attach to db2prod1
    db2 get dbm cfg show detail | egrep -i "diaglevel|diagpath"

    Also what happens if you do this:

    db2diag -A
    db2 update dbm cfg using diaglevel 4
    Wait a while...
    check to see if anything is added.

    What happens if you move it, albeit temporarily:

    db2 update dbm cfg using diagpath /.../.../
    ( where is is a different path that db2prod1 has rw access too )

    If we still don't get a db2diag.log then I'd used strace of the db2sycs PID to see what problems it is reporting in accessing the file.

    Stephen.
  • starndawn
    starndawn
    37 Posts

    Re: failed to open db2diag.log

    ‏2012-11-21T03:24:32Z  
    • mwandishi
    • ‏2012-11-20T11:13:25Z
    So we know that:

    db2diag.log is in the correct place, the permissions & ownerships are correct, DIAGLEVEL is default and db2 is up meaning we should be getting some messages. However, it is 0 bytes and we have a 'disk error'.

    I assume there is sufficient disk space in the path where it resides ?

    Can you touch a new file in there ( touch test.log in /db2home/db2prod1/sqllib/db2dump as the instance owner ) what happens ?

    Also, I wonder if there is a delayed value for DIAGLEVEL / DIAGPATH, can you try this:

    db2 attach to db2prod1
    db2 get dbm cfg show detail | egrep -i "diaglevel|diagpath"

    Also what happens if you do this:

    db2diag -A
    db2 update dbm cfg using diaglevel 4
    Wait a while...
    check to see if anything is added.

    What happens if you move it, albeit temporarily:

    db2 update dbm cfg using diagpath /.../.../
    ( where is is a different path that db2prod1 has rw access too )

    If we still don't get a db2diag.log then I'd used strace of the db2sycs PID to see what problems it is reporting in accessing the file.

    Stephen.
    Hi Stephen,

    Thank you for your suggestion!

    The log file cannot be written, it reported "Operation not permitted". We used lsattr/chattr to change the hidden file permission of this log file, and removed it successfully. Now it is ok for db2 instance to write messages into this file.
  • mwandishi
    mwandishi
    47 Posts

    Re: failed to open db2diag.log

    ‏2012-11-21T09:50:22Z  
    • starndawn
    • ‏2012-11-21T03:24:32Z
    Hi Stephen,

    Thank you for your suggestion!

    The log file cannot be written, it reported "Operation not permitted". We used lsattr/chattr to change the hidden file permission of this log file, and removed it successfully. Now it is ok for db2 instance to write messages into this file.
    Hi,

    No problem and you are welcome. Glad it is working normally again.

    Stephen.