IBM Support

Diagnosing no data within a workspace

Technical Blog Post


Abstract

Diagnosing no data within a workspace

Body

First look to see if there is an error in the view:

  • Return Codes 202 / 209 errors or 67109066 / 67109073  (these are for remote TEMS).  This means there is an application support issue. You can use ITMSUPER or the Self Monitoring workspaces provided in 622 and above.
  • Return Code 5 implies something was not there like a managed system.  Verify the query distribution.
     
  • Return Code 3000/3001 implies historical data is not available yet.  Verify that data has been captured by checking for the binary files.
     
  • Return Code 155 means an agent did not respond in time.  This is a timeout condition.
     
  • If there is no error, then this is most likely a query timeout or the agent data collector has hung.

To verify consider setting the following traces:

  • TEPS: 
    • ERROR (UNIT:CTSQL INPUT OUTPUT STATE ERROR)  (UNIT:CTDataBus INPUT ERROR) (UNIT:CTReport INPUT,ERROR) (UNIT:CTRASHelper INPUT,ERROR)    (UNIT:CTCMWManagedSystemsManager ALL)
  • TEMS:
    • ERROR (UNIT:KDSSQPRS INPUT OUTPUT STATE ERROR) (UNIT:KDSSQRUN INPUT OUTPUT STATE ERROR) (UNIT:KPXRPCRQ INPUT OUTPUT STATE ERROR) (UNIT:KPXREQDS INPUT OUTPUT STATE ERROR) 
  • Agent:  
    • ERROR (UNIT:KRA INPUT OUTPUT STATE ERROR) (UNIT:KRAAFIRA ALL) (UNIT:Kpc ALL)    - where pc is the product code for the agent.  Ex. UX for Unix, NT for Windows.

       

Following are some log snipnets which show the lack of data due to an agent data collector hang:

  • TEPS log (This is the request being sent off to get the data)

    (2010.286 03.23.03-23:ctsqlstatement.cpp,217,"SQLStatement::SQLStatement")
    HUB_IBM(48): SELECT UNIXPS.ORIGINNODE, UNIXPS.UCMD, UNIXPS.VSIZE, UNIXPS.PID, UNIXPS.PPID, UNIXPS.UID, UNIXPS.CPU, UNIXPS.CPUPERCENT, UNIXPS.SIZE, UNIXPS.MEMPERCENT, UNIXPS.NICE, UNIXPS.PRIORITY, UNIXPS.EXECSTATE, UNIXPS.STARTTIME, UNIXPS.TIME, UNIXPS.TTY, UNIXPS.EVENT, UNIXPS.FLAG, UNIXPS.TIMESTAMP, UNIXPS.REPTYPE, UNIXPS.PGID FROM OMUNX.UNIXPS AT( 'HUB_IBM1' ) WHERE (UNIXPS.ORIGINNODE = ?) AND LAST(10) AND SYSTEM.PARMA("TIMEOUT","600",3) ORDER BY UNIXPS.VSIZE ASC  
    (2010.28603.23.03-23:ctsqlstatement.cpp,242,"SQLStatement::SQLStatement") HUB_IBM(48): Values: 'IBMUNIX1:KUX'

     
  • TEMS log                                                                             

    (2010.286 03.23.03-12:kpxreqds.cpp,2707,"ConvertDtypeToString") input parms: dtype 452, len 10                                                 
    (2010.286 03.23.03-12:kpxreqds.cpp,1443,"buildThresholdsOldStyle")           
    IndexFilter: (seq=1/op=1/type=452/'ORIGINNODE'/0x40000000/len=10) value: 'IBMUNIX1:KUX'   
       
    FYI - You will not always see the SQL generated because it is parmmarkered SQL(s) that uses ORIGINNODE = ?.  It may have been  previously "complied" when IBMUNIX2:KUX called it.
     
  • Agent log we see the UNIXPS data collector is hung:
     
    (2010.286 03.23.03-17:kraafira.cpp,776,"DriveDataCollection") OMUNX.UNIXPS, <835715699,1376781261>  expired.                       
    (2010.286 03.23.03-17:kux03agt.cpp,181,"TakeSample") Active RAS1 Classes: EVERYT EVERYE EVERYU                                 
    (2010.286 03.23.03-17:kux03agt.cpp,181,"TakeSample") Entry                   
    (2010.286 03.23.03-18:kraarpcx.cpp,207,"IRA_NCS_Thresholds_from_xmit_rep") Thresholds: type <1>, planLen <0>, litLen <78>, lastByte <0>                 
    (2010.286 03.23.03-18:kraafmgr.cpp,713,"Start") Start received  <812647032,0> on *.UNIXPS                                                    
    (2010.286 03.23.03-18:kraafmgr.cpp,717,"Start") Value of instruction <>      
    (2010.286 03.23.03-18:kux03agt.cpp,463,"TakeSampleConstructor") Entry        
    (2010.286 03.23.03-18:kux03agt.cpp,491,"TakeSampleConstructor") reptye from kux03agt      pid 0 ppid 0 pgid 0                   
    (2010.286 03.23.03-18:kux03agt.cpp,493,"TakeSampleConstructor") Exit         
    (2010.286 03.23.03-18:kraafmgr.cpp,745,"Start") Start complete  <812647032,1378878412> on *.UNIXPS, status = 0                      
    (2010.286 03.23.03-17:kux03agt.cpp,205,"TakeSample") Obtaining ps data for node: IBMUNIX1:KUX
    (2010.286 03.23.03-17:kux03agt.cpp,214,"TakeSample") the process report type is                                                                      
    We can tell this is a hang because thread 17 never ends.  The TakeSample function does not exit for more than 15 mintues.  At this point no data from this agent will be returned.

 

What you should see:

  • TEPS log

    (2010.307 17.11.27-28E4:ctsqlstatement.cpp,217,"SQLStatement::SQLStatement") HUB_IBM-C982C88B1C6(42): SELECT WTPHYSDSK.ORIGINNODE, WTPHYSDSK.INSTCNAME, WTPHYSDSK.DISKNUMBER, WTPHYSDSK.DSKQUELEN, WTPHYSDSK.AVGQLEN, WTPHYSDSK.AVGRDQLEN, WTPHYSDSK.AVGWRQLEN, WTPHYSDSK.PCTDSKTIME, WTPHYSDSK.PCTDSKRDTM, WTPHYSDSK.PCTDSKWRTM, WTPHYSDSK.PCTDSKIDTM, WTPHYSDSK.DSKTRNSSEC, WTPHYSDSK.DSKRDSEC, WTPHYSDSK.DSKWRTSEC, WTPHYSDSK.DSKBTSSEC8, WTPHYSDSK.DSKRDBTSE8, WTPHYSDSK.DSKWRTBYT8, WTPHYSDSK.AVDSKBYTT8, WTPHYSDSK.AVGDSKBYT8, WTPHYSDSK.AVDSKBYTW8, WTPHYSDSK.AVDSKSECTF, WTPHYSDSK.AVDSKSECRD, WTPHYSDSK.AVDSKSECWT, WTPHYSDSK.TIMESTAMP FROM KNT.WTPHYSDSK AT( 'HUB_IBM-C982C88B1C6' ) WHERE (WTPHYSDSK.ORIGINNODE = ?) AND SYSTEM.PARMA("TIMEOUT","600",3)
    2010.307 17.11.27-28E4:ctsqlstatement.cpp,242,"SQLStatement::SQLStatement") HUB_IBM-C982C88B1C6(42): Values: 'Primary:IBM-C982C88B1C6:NT'
    (2010.307 17.11.27-1CE4:ctsqlaccesssql1.cpp,1015,"CTSQLEvaluatorSQL1_i::AccessElement::pullSequenceWithTimeout") HUB_IBM-C982C88B1C6(42): Rows returned: 2
     
  • TEMS log

    (2010.307 17.11.27-A00:kdssqprs.c,643,"PRS_ParseSql") SELECT WTPHYSDSK.ORIGINNODE, WTPHYSDSK.INSTCNAME, WTPHYSDSK.DISKNUMBER,
    (2010.307 17.11.27-A00:kdssqprs.c,643,"PRS_ParseSql") WTPHYSDSK.DSKQUELEN, WTPHYSDSK.AVGQLEN, WTPHYSDSK.AVGRDQLEN, WTPHYSDSK.A
    (2010.307 17.11.27-A00:kdssqprs.c,643,"PRS_ParseSql") VGWRQLEN, WTPHYSDSK.PCTDSKTIME, WTPHYSDSK.PCTDSKRDTM, WTPHYSDSK.PCTDSKWR
    (2010.307 17.11.27-A00:kdssqprs.c,643,"PRS_ParseSql") TM, WTPHYSDSK.PCTDSKIDTM, WTPHYSDSK.DSKTRNSSEC, WTPHYSDSK.DSKRDSEC, WTPH
    (2010.307 17.11.27-A00:kdssqprs.c,643,"PRS_ParseSql") YSDSK.DSKWRTSEC, WTPHYSDSK.DSKBTSSEC8, WTPHYSDSK.DSKRDBTSE8, WTPHYSDSK.D
    (2010.307 17.11.27-A00:kdssqprs.c,643,"PRS_ParseSql") SKWRTBYT8, WTPHYSDSK.AVDSKBYTT8, WTPHYSDSK.AVGDSKBYT8, WTPHYSDSK.AVDSKBY
    (2010.307 17.11.27-A00:kdssqprs.c,643,"PRS_ParseSql") TW8, WTPHYSDSK.AVDSKSECTF, WTPHYSDSK.AVDSKSECRD, WTPHYSDSK.AVDSKSECWT, W
    (2010.307 17.11.27-A00:kdssqprs.c,643,"PRS_ParseSql") TPHYSDSK.TIMESTAMP FROM KNT.WTPHYSDSK AT( 'HUB_IBM-C982C88B1C6' ) WHERE
    (2010.307 17.11.27-A00:kdssqprs.c,643,"PRS_ParseSql") (WTPHYSDSK.ORIGINNODE = ?) AND SYSTEM.PARMA("TIMEOUT","600",3)
    (2010.307 17.11.27-A00:kdssqprs.c,4225,"PRS_BuildNumericElem") 32 bit   Signed Integer           3 len=4 type=17
    (2010.307 17.11.27-A00:kdssqprs.c,1112,"PRS_DestroySql") 179 Mallocs
    (2010.307 17.11.27-A00:kdssqprs.c,1113,"PRS_DestroySql") 181 Frees
    (2010.307 17.11.27-2E24:kpxreqds.cpp,2707,"ConvertDtypeToString") input parms: dtype 452, len 26
    (2010.307 17.11.27-2E24:kpxreqds.cpp,1443,"buildThresholdsOldStyle") IndexFilter: (seq=1/op=1/type=452/'ORIGINNODE'/0x40000000/len=26) value: 'Primary:IBM-C982C88B1C6:NT'
    (2010.307 17.11.27-1E98:kpxrpcrq.cpp,749,"IRA_NCS_Sample") Rcvd 2 rows sz 284 tbl *.WTPHYSDSK req  <1519390290,1185941329> node <Primary:IBM-C982C88B1C6:NT
     
  • Agent log

    (2010.307 17.11.26-149C:kraafira.cpp,766,"DriveDataCollection") Entry
    (2010.307 17.11.26-149C:kraafira.cpp,776,"DriveDataCollection") KNT.WTPHYSDSK, <1519390290,1185941329>  expired.
    (2010.307 17.11.26-149C:knt09agt.cpp,139,"TakeSample") Entry
    (2010.307 17.11.26-149C:kntcoi.cpp,145,"KntKountInterface") Entry
    (2010.307 17.11.26-149C:kntcoi.cpp,460,"KntKountInterface") Exit: 0x0
    (2010.307 17.11.26-149C:knt09agt.cpp,192,"TakeSample") Perfmon values --> Disk_Name=0 C:                                                            , 
    +2010.307 17.11.26 _Disk_Time=26, _Disk_Idle_Time=79, _Disk_Read_Time=21, _Disk_Write_Time=4
    (2010.307 17.11.26-149C:knt09agt.cpp,262,"TakeSample") Updated values --> Disk_Name=0 C:                                                            ,
    +2010.307 17.11.26 _Disk_Time=21, _Disk_Idle_Time=79, _Disk_Read_Time=16, _Disk_Write_Time=3,
    (2010.307 17.11.26-149C:knt09agt.cpp,192,"TakeSample") Perfmon values --> Disk_Name=_Total                                                          , 
    +2010.307 17.11.26 _Disk_Time=26, _Disk_Idle_Time=79, _Disk_Read_Time=21, _Disk_Write_Time=4
    (2010.307 17.11.26-149C:knt09agt.cpp,262,"TakeSample") Updated values --> Disk_Name=_Total                                                          ,
    +2010.307 17.11.26 _Disk_Time=21, _Disk_Idle_Time=79, _Disk_Read_Time=16, _Disk_Write_Time=3,
    (2010.307 17.11.26-149C:kraafira.cpp,3743,"CheckDistributionList") Entry
    (2010.307 17.11.26-149C:kraafira.cpp,3794,"CheckDistributionList") Exit: 0x1
    (2010.307 17.11.26-149C:kraafira.cpp,1002,"InsertRow") rowsize = 284, newsize = 20, newbytes = 5680, _allocated = 0, _allocSize = 20
    (2010.307 17.11.26-149C:kraafira.cpp,3743,"CheckDistributionList") Entry
    (2010.307 17.11.26-149C:kraafira.cpp,3794,"CheckDistributionList") Exit: 0x1
    (2010.307 17.11.26-149C:knt09agt.cpp,296,"TakeSample") Exit
    (2010.307 17.11.26-149C:krant09b.cpp,331,"PrintSelf") knt_wtphysdsk_base @4896858 <1519390290,1185941329>
    (2010.307 17.11.26-149C:krant09b.cpp,333,"PrintSelf")  ::appl=KNT,table=WTPHYSDSK,interval=0,samptype=3
    (2010.307 17.11.26-149C:kraaprdf.cpp,601,"PrintSelf")  Predicates {
    (2010.307 17.11.26-149C:kraaprdf.cpp,618,"PrintSelf")    <ORIGINNODE,1,4897368<64>Primary:IBM-C982C88B1C6:NT                                      ,8>
    (2010.307 17.11.26-149C:kraaprdf.cpp,626,"PrintSelf") <0x4897368,0x40>
    +2010.307 17.11.26     00000000   5072696D 6172793A  49424D2D 43393832   Primary:IBM-C982
    +2010.307 17.11.26     00000010   43383842 3143363A  4E542020 20202020   C88B1C6:NT......
    +2010.307 17.11.26     00000020   20202020 20202020  20202020 20202020   ................
    +2010.307 17.11.26     **************  00000030 - 00000040: Same as above  **************
    (2010.307 17.11.26-149C:kraaprdf.cpp,618,"PrintSelf")    <PARMA,0,48973D0<3>600,0>
    (2010.307 17.11.26-149C:kraaprdf.cpp,626,"PrintSelf") <0x48973D0,0x3>
    +2010.307 17.11.26     00000000   363030                                 600            
    (2010.307 17.11.26-149C:kraaprdf.cpp,633,"PrintSelf")  }
    (2010.307 17.11.26-149C:krant09b.cpp,339,"PrintSelf")  _data {
    (2010.307 17.11.26-149C:krant09b.cpp,340,"PrintSelf")   count=2, allocated=20, allocSize=20
    (2010.307 17.11.26-149C:krant09b.cpp,345,"PrintSelf")   {
    (2010.307 17.11.26-149C:krant09b.cpp,346,"PrintSelf")   originnode="Primary:IBM-C982C88B1C6:NT   
    (2010.307 17.11.26-149C:krant09b.cpp,382,"PrintSelf") }
    (2010.307 17.11.26-149C:kraafira.cpp,905,"DriveDataCollection") Exit: 0x0
    (2010.307 17.11.26-149C:kraadspt.cpp,706,"sendDataToProxy") Sending 2 rows for  KNT.WTPHYSDSK, <1519390290,1185941329>.

 

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"","label":""},"Component":"","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"","label":""}}]

UID

ibm11083447

Page Feedback