Topic
  • 6 replies
  • Latest Post - ‏2010-11-04T01:53:47Z by SystemAdmin
eyley
eyley
6 Posts

Pinned topic Region crash when using multi-region environments. - Need any help...

‏2010-08-13T01:47:40Z |
I have made 13 regions on AIX 6.1. TL5.
TXseries version is 7.1 and fixpack 7.1.0.2 applied.

Till 4 regions, no crash.
But serveral decades of minutes passed after starting all 13 regions, some of regions were crashed. Now only 9 regions are alive, 4 regions gone.

The message is
========================
ERZ057002E/0136 08/12/10 18:19:08.094028572 F81TAR21 12845234/0001 : CICS i
nternal error: Unsuccessful condition '(!LogInUse)' for function 'DamJO_LogDaemo
n ' on line 410 in file '/cics/FSB/src/dam/jo/src/damjo_logd.c' of module 24
ERZ057005E/0136 08/12/10 18:19:08.098259009 F81TAR21 12845234/0001 : CICS i
nternal error: Abnormally terminating region with exit code 'U5701'
ERZ010003I/0094 08/12/10 18:19:08.098429623 F81TAR21 12845234/0001 : CICS i
s performing region abnormal termination in process 'cicsld'
================================
And dumps was created and regions gone.

Anyone can talk about LogDaemon?
What actions needed?

I just requested the PMR process, but need more speedy answer.
Anybody who can help me!! Please...
Updated on 2010-11-04T01:53:47Z at 2010-11-04T01:53:47Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    308 Posts

    Re: Region crash when using multi-region environments. - Need any help...

    ‏2010-08-13T05:02:11Z  
    Unfortunately the error information you have pasted in the post is too little to understand the problem. Would you be able to attach the console.<nnnnnn> and symrecs.<nnnnnn> files in this forum?. In general the logs are used by the application servers to store the transactional records and these logs are managed by the LogDaemon. There would be multiple logs (which you would find under the region's log directory) as there will be multiple application servers. Briefly, from the error it does look like that the application server is not able to open the log file, as it is already in use. Usually the logs are identified by the Application server ID (which starts from 101, 102, 103, ... and so on). So one way to check is to see in the console log if there are multiple application servers started with the same Application Server Id (a situation that shouldn't occur actually!!!)...then this can result in this error.

    In any case if you attach the console log, I can take a quick look.

    Cheers,
    Hari
  • eyley
    eyley
    6 Posts

    Re: Region crash when using multi-region environments. - Need any help...

    ‏2010-08-13T05:43:44Z  
    Unfortunately the error information you have pasted in the post is too little to understand the problem. Would you be able to attach the console.<nnnnnn> and symrecs.<nnnnnn> files in this forum?. In general the logs are used by the application servers to store the transactional records and these logs are managed by the LogDaemon. There would be multiple logs (which you would find under the region's log directory) as there will be multiple application servers. Briefly, from the error it does look like that the application server is not able to open the log file, as it is already in use. Usually the logs are identified by the Application server ID (which starts from 101, 102, 103, ... and so on). So one way to check is to see in the console log if there are multiple application servers started with the same Application Server Id (a situation that shouldn't occur actually!!!)...then this can result in this error.

    In any case if you attach the console log, I can take a quick look.

    Cheers,
    Hari
    Hi Hari,
    Thank you for keen attention.

    in RD, min server=1, max server=30.

    ========================
    :ml14
    ResourceDescription="Region Definition"
    AmendCounter=15
    Modifiable=yes
    StartType=cold
    Groups=
    StartupProgList=""
    ShutdownProgList1=""
    ShutdownProgList2=""
    DefaultUserId="CICSUSER"
    FileSystemType=SFS
    RDBMSInstance=""
    FileRSLCheck=external
    TransientDataRSLCheck=external
    TemporaryStorageRSLCheck=external
    JournalRSLCheck=external
    ProgramRSLCheck=external
    TransactionRSLCheck=external
    ESMLoad=no
    ESMModule=""
    ReleaseNum="0710"
    LocalSysId="FA15"
    LocalNetworkName="KRIBKKB1"
    CWASize=512
    MinServer=1
    MaxServer=30
    ClassMaxTasks=1,1,1,1,1,1,1,1,1,1
    ClassMaxTaskLim=0,0,0,0,0,0,0,0,0,0
    ServerIdleLimit=300
    SafetyLevel=none
    RPCListenerThreads=0
    DateForm=ddmmyy
    ClassTableSize=5,50,50,1,50,50,50,20,5,50,1,20,5,5,20
    MaxRegionPool=31457280
    MaxTaskPrivatePool=10485760
    MaxTSHPool=10485760
    RegionPoolThreshold=90
    TaskShPoolThreshold=90
    TaskSHNumBuckets=512
    LoadDataNumBuckets=512
    SysDump=no
    PCDump=yes
    ABDump=yes
    DumpName="dumps"
    CoreDumpName="dir1"
    TraceDirectorySystem="/var/cics_regions/%R/dumps/dir1"
    TraceDirectoryUser="/var/cics_servers/archives"
    TraceFlagAux=off
    TraceFlagBuffer=on
    TraceFlagExec=off
    TraceFlagExternal=off
    TraceFlagMaster=off
    TraceFlagSystem=off
    TraceFlagUser=off
    TraceMaxSizeAux=0
    TraceMaxSizeBuffer=131072
    TraceSystemSpec="all=4"
    IntrospectInterval=30
    IntrospectLevel=minimal
    ISCDelayMinutes=30
    CUBSDelayMinutes=30
    CARPDelayHours=8
    ProtectPurgeDelayPeriod=8
    PurgeDelayPeriod=8
    StatsRecord=yes
    StatFile="statsfile"
    SufficesSupported=yes
    CheckpointInterval=10000
    DefaultFileServer="/.:/cics/sfs/upafep81"
    RecTSQFile="F81TAR21cicsrectsqfile"
    RecTSQIndex="cicsrectsqidx"
    RecTSQVol="sfs_%S"
    RecTSQPrePages=5
    RecTSQMaxRecs=1000000
    NonRecTSQFile="F81TAR21cicsnrectsqfil"
    NonRecTSQIndex="cicsnrectsqidx"
    NonRecTSQVol="sfs_%S"
    NonRecTSQPrePages=5
    NonRecTSQMaxRecs=1000000
    LogicalTDQFile="F81TAR21cicstdqlgfile"
    LogicalTDQIndex="cicstdqlgidx"
    LogicalTDQVol="sfs_%S"
    LogicalTDQPrePages=5
    LogicalTDQMaxRecs=1000000
    PhysicalTDQFile="F81TAR21cicstdqphfile"
    PhysicalTDQIndex="cicstdqphidx"
    PhysicalTDQVol="sfs_%S"
    PhysicalTDQPrePages=5
    PhysicalTDQMaxRecs=1000000
    NonRecTDQFile="F81TAR21cicstdqnofile"
    NonRecTDQIndex="cicstdqnoidx"
    NonRecTDQVol="sfs_%S"
    NonRecTDQPrePages=5
    NonRecTDQMaxRecs=1000000
    LocalQProtectFile="F81TAR21cicsplqfile"
    LocalQProtectIndex="cicsplqidx"
    LocalQProtectVol="sfs_%S"
    LocalQProtectPrePages=5
    LocalQProtectMaxRecs=1000000
    LocalQFile="F81TAR21cicsnlqfile"
    LocalQIndex="cicsnlqidx"
    LocalQVol="sfs_%S"
    LocalQPrePages=5
    LocalQMaxRecs=1000000
    TSQAgeLimit=20
    ProgramCacheSize=0
    LocalLUName="TFEP15"
    RegionPoolBase=2684354560
    TaskSharedPoolBase=2952790016
    ServerSideTran=no
    AllowDebugging=no
    HTMLBrowser=""
    AutoinstallMode=default
    MaxConsoleSize=0
    MaxTaskCPU=0
    MaxTaskCPUAction=warning
    TransDumpTrace=no
    PostScanInterval=0
    IIOPObjrefsVersion="1.1"
    IIOPRequestTimeout=100
    IIOPGroupUUID=""
    IIOPGroupHost=""
    IIOPGroupPort=""
    IIOPGroupSSLPort=""
    XPRecvTimeout=0
    TDPurgeInterval=0
    EAMLoad=no
    EAMModule=""
    ServerMemCheckInterval=0
    ServerMemCheckLimit=0
    WLMListenerList="{0.0.0.0,0}"
    WLMMinServer="{0}"
    WLMMaxServer="{0}"
    WLMWillingness="{-1}"
    WLMResponseTimeOut=10
    EnableAppProbe=no
    EnableTaskInfo=no
    DumpRemoteInfo=no
    =============================

    log directory hold 101 ~ 104 AS content and control files.
    Are those necessary also?

    Thank you again, Hari!
  • eyley
    eyley
    6 Posts

    Re: Region crash when using multi-region environments. - Need any help...

    ‏2010-08-13T06:21:01Z  
    Unfortunately the error information you have pasted in the post is too little to understand the problem. Would you be able to attach the console.<nnnnnn> and symrecs.<nnnnnn> files in this forum?. In general the logs are used by the application servers to store the transactional records and these logs are managed by the LogDaemon. There would be multiple logs (which you would find under the region's log directory) as there will be multiple application servers. Briefly, from the error it does look like that the application server is not able to open the log file, as it is already in use. Usually the logs are identified by the Application server ID (which starts from 101, 102, 103, ... and so on). So one way to check is to see in the console log if there are multiple application servers started with the same Application Server Id (a situation that shouldn't occur actually!!!)...then this can result in this error.

    In any case if you attach the console log, I can take a quick look.

    Cheers,
    Hari
    Hari,

    As you said, I saw the cicsas 101 were started twice in console message.
    It may be problem?

    Thanks,
  • SystemAdmin
    SystemAdmin
    308 Posts

    Re: Region crash when using multi-region environments. - Need any help...

    ‏2010-08-13T06:30:46Z  
    • eyley
    • ‏2010-08-13T06:21:01Z
    Hari,

    As you said, I saw the cicsas 101 were started twice in console message.
    It may be problem?

    Thanks,
    Ok... this confirms my understanding. This looks to be a known issue and has been fixed in 7.1.0.2 level. You can find the fix details here:

    http://www-933.ibm.com/support/fixcentral/swg/quickorder?parent=ibm~WebSphere&product=ibm/WebSphere/TXSeries+for+Multiplatforms&release=7.1.0.2&platform=AIX&function=fixId&fixids=7.1.0.2-TXSeries-AIX-InterimServiceFix2&includeRequisites=1&downloadMethod=http&source=fc

    TXSeries 7.1.0.2 InterimServiceFix 2

    This patch contains fixes for the following :
    APAR No Defect Abstract


    IZ76269 210342 PUT STATEMENT ON PL/I RESULTS IN ERZ027001E .
    IZ80091 210490 PL/I program doesn't log ProgaName to CPLI.out during exception
    IZ81023 210524 U5701 abend caused due to duplicate Application server Id
    IZ81026 210527 CICS-PL/I generates invalid code for EXEC CICS statements

    (C) Copyright International Business Machines Corporation, 2005. All Rights Reserved.
    <<End_of_APAR_Details>>
    HTH.

    Cheers,
    Hari
  • eyley
    eyley
    6 Posts

    Re: Region crash when using multi-region environments. - Need any help...

    ‏2010-08-16T00:53:23Z  
    Ok... this confirms my understanding. This looks to be a known issue and has been fixed in 7.1.0.2 level. You can find the fix details here:

    http://www-933.ibm.com/support/fixcentral/swg/quickorder?parent=ibm~WebSphere&product=ibm/WebSphere/TXSeries+for+Multiplatforms&release=7.1.0.2&platform=AIX&function=fixId&fixids=7.1.0.2-TXSeries-AIX-InterimServiceFix2&includeRequisites=1&downloadMethod=http&source=fc

    TXSeries 7.1.0.2 InterimServiceFix 2

    This patch contains fixes for the following :
    APAR No Defect Abstract


    IZ76269 210342 PUT STATEMENT ON PL/I RESULTS IN ERZ027001E .
    IZ80091 210490 PL/I program doesn't log ProgaName to CPLI.out during exception
    IZ81023 210524 U5701 abend caused due to duplicate Application server Id
    IZ81026 210527 CICS-PL/I generates invalid code for EXEC CICS statements

    (C) Copyright International Business Machines Corporation, 2005. All Rights Reserved.
    <<End_of_APAR_Details>>
    HTH.

    Cheers,
    Hari
    Thank you Hari,

    I've received the interim fix and monitor it.

    Great Thanks to you...
  • SystemAdmin
    SystemAdmin
    308 Posts

    Re: Region crash when using multi-region environments. - Need any help...

    ‏2010-11-04T01:53:47Z  
    Hari77 wrote:
    Unfortunately the error information you have pasted in the post is too little to understand the problem. Would you be able to attach the console.<nnnnnn> and symrecs.<nnnnnn> files in this forum?. In general the logs are used by the application servers to store the transactional records and these logs are managed by the LogDaemon. There would be multiple logs (which you would find under the region's log directory) as there will be multiple application servers. Briefly, from the error it does look like that the application server is not able to open the log file, as it is already in use. Usually the logs are identified by the Application server ID (which starts from 101, 102, 103, ... and so on). So one way to check is to see in the console log if there are multiple application servers started with the same Application Server Id (a situation that shouldn't occur actually!!!)...then this can result in this error.

    In any case if you attach the console log, I can take a quick look.

    Cheers,
    Hari</nnnnnn></nnnnnn>

    Many thanks to your description! This is what I'm looking for.