Topic
  • 9 replies
  • Latest Post - ‏2013-06-10T14:24:03Z by IvanHargreaves
yairbar
yairbar
20 Posts

Pinned topic CWWKE0064E - Need advice: Liberty profile failure under CICS 5.1 Developer

‏2013-04-24T12:28:25Z |

Hello,

When enabling JVM  using profile DFHWLP for Web container w/ Liberty profile  under CICS 5.1 developer trial,   it fails to start,  saying at the error log :

CWWKE0064E  unable to load zOS native library  from (my work-dir:) /usr/lpp/cicsts/cicsts51/dvlp/wlp/lib/native/zos/s390x/libzNativeServices.so.

Such library realiy  does not exist in my USS, but why he looks for zOS native when unde CICS ? Is it correct?

If no, how do I prevent him from searching it ? If yes, how do I get this library ?

 

Appreciate your help,

Yair

Updated on 2013-04-25T09:06:50Z at 2013-04-25T09:06:50Z by yairbar
  • Adam Coulthard
    Adam Coulthard
    13 Posts

    Re: CWWKE0064E - Need advice: Liberty profile failure under CICS 5.1 Developer

    ‏2013-05-09T15:28:35Z  

    Hello Yair,

    This message can be safely ignored and does not impact the starting of the Liberty profile.  What other messages are you seeing within the STDOUT / STDERR / CICS joblog?  Hopefully these will help to get to the bottom of the issue.

    Cheers

    Adam Coulthard

  • yairbar
    yairbar
    20 Posts

    Re: CWWKE0064E - Need advice: Liberty profile failure under CICS 5.1 Developer

    ‏2013-06-03T07:32:34Z  

    Hello Yair,

    This message can be safely ignored and does not impact the starting of the Liberty profile.  What other messages are you seeing within the STDOUT / STDERR / CICS joblog?  Hopefully these will help to get to the bottom of the issue.

    Cheers

    Adam Coulthard

    Thanks, Adam !

    (pace of posts exchange in this forum is quite slow, so I had not visited frequently, sorry...)

    Actually   my  problem with the Liberty JVMServer is much more serious.

    It fails to become Enabled, reporting a "timeout" which is NOT customizable (i.e. at least  not under control of OSGI_FRAMEWORK_TIMEOUT in DFH$WLP profile).

    If you are interested, please see details in  the attched JVMserver's logs  and CICS diagnostics.

    It looks to me that this  IBM's product is absolutely unstable and immature at this phase, and trouble shooting info quite unfriendly.

     

    Regards,

    Yair Barzilai

     

     

  • Adam Coulthard
    Adam Coulthard
    13 Posts

    Re: CWWKE0064E - Need advice: Liberty profile failure under CICS 5.1 Developer

    ‏2013-06-03T07:40:31Z  

    Hi Yair,

    Thanks for the diagnostics we'll have a look into it and get back to you.

    Cheers

    Adam Coulthard

  • yairbar
    yairbar
    20 Posts

    Re: CWWKE0064E - Need advice: Liberty profile failure under CICS 5.1 Developer

    ‏2013-06-03T10:58:43Z  

    Hi Yair,

    Thanks for the diagnostics we'll have a look into it and get back to you.

    Cheers

    Adam Coulthard

    Thanks, appreciate it very much...

    So I am waiting for your comments.

     

    Reards,

    Yair B.

  • Adam Coulthard
    Adam Coulthard
    13 Posts

    Re: CWWKE0064E - Need advice: Liberty profile failure under CICS 5.1 Developer

    ‏2013-06-03T13:13:48Z  
    • yairbar
    • ‏2013-06-03T10:58:43Z

    Thanks, appreciate it very much...

    So I am waiting for your comments.

     

    Reards,

    Yair B.

    Hi Yair,

    We've been looking at the diagnostics that you sent us and we have a couple more questions regarding your setup.

    1. Have you got all the latest service for zFS applied to your system?
    2. Are you using shared classcache?
    3. Are you using the autoconfigure liberty option? or are you using a custom server.xml
    4. How many JVMServers with Liberty enabled are you starting up at once?
    5. Would it be possible to see your JVMProfile? You can email it directly to me if you would prefer (a.p.coulthard at uk.ibm.com)

    Cheers

    Adam Coulthard

  • yairbar
    yairbar
    20 Posts

    Re: CWWKE0064E - Need advice: Liberty profile failure under CICS 5.1 Developer

    ‏2013-06-03T14:06:25Z  

    Hi Yair,

    We've been looking at the diagnostics that you sent us and we have a couple more questions regarding your setup.

    1. Have you got all the latest service for zFS applied to your system?
    2. Are you using shared classcache?
    3. Are you using the autoconfigure liberty option? or are you using a custom server.xml
    4. How many JVMServers with Liberty enabled are you starting up at once?
    5. Would it be possible to see your JVMProfile? You can email it directly to me if you would prefer (a.p.coulthard at uk.ibm.com)

    Cheers

    Adam Coulthard

    Hello Adam,

    Attached my most recent JVM Profile.

    I run under a newly installed zOS 1.13, believe zFS is OK but i'll double check it with sys.

    I  run a single Liberty JVMServer, no shared classcache, using a  'server.xml'  which was generated once by autoconfigure = true and then supposed to be re- used  each time (i.e. profile is now set to autoconfigure = false for a "server" called "my3rdServer").  Am I right ?

    My complaint  is , never mind what would be  my config error (I presume...),is  that the TIMEOUT in the log looks to me  neither explained nor customizable....

    Thanks!

    Yair

     

     

    Attachments

  • IvanHargreaves
    IvanHargreaves
    27 Posts

    Re: CWWKE0064E - Need advice: Liberty profile failure under CICS 5.1 Developer

    ‏2013-06-03T15:01:01Z  
    • yairbar
    • ‏2013-06-03T14:06:25Z

    Hello Adam,

    Attached my most recent JVM Profile.

    I run under a newly installed zOS 1.13, believe zFS is OK but i'll double check it with sys.

    I  run a single Liberty JVMServer, no shared classcache, using a  'server.xml'  which was generated once by autoconfigure = true and then supposed to be re- used  each time (i.e. profile is now set to autoconfigure = false for a "server" called "my3rdServer").  Am I right ?

    My complaint  is , never mind what would be  my config error (I presume...),is  that the TIMEOUT in the log looks to me  neither explained nor customizable....

    Thanks!

    Yair

     

     

    Hi Yair,

    The OSGI_FRAMEWORK_TIMEOUT value is not related to the timeout message you see. The controller timeout can be configured by a separate 'IBM service only' configuration setting. It is not documented because it should only be changed under IBM guidance, and in your case I don't think it is relevant or useful to change.

    Back to the failure message, this indicates that the CICS-feature of the embedded Liberty instance could not be started within a 'reasonable' amount of time. There could be so many reasons for that, and we may require a CICS trace and a dfhjvmtrc to determine the underlying problem. I suggest you set CICS trace of AP and SJ domains to ALL before you start the JVMSERVER. Then provide the CICS trace dataset, CICS joblog, and the zFS equivalents (dfhjvmtrc, dfhjvmerr, dfhjvmout) if none of the following actions help.

    I notice the value in your JVM profile:  WORK_DIR=/usr/lpp/cicsts/cicsts51/dvlp. This WORK_DIR needs to allow read/write access; while your zFS path looks like a location we would normally expect to be read only.

    I also notice you are not using the autoconfigure option. However you say you used it initially. The autoconfigure option will need to be used initially for each Liberty 'server' you start. So for example, autoconfigure of myserver1 does not mean that a name change to myserver2 will work. I say this, just in case you have changed server name without using autoconfigure for that server.

    If ensuring the WORK_DIR is read/write does not cure the problem, nor does the autoconfigure option, then it may be a genuine timeout problem. Indications of this would be:

    - you are starting many JVMSERVER Liberty servers at the same time.

    - there was a zFS i/o defect that caused very slow zFS response, I forget the details, but please ensure you have the latest service to z/OS.

    For the case where you are starting many servers simultaneously, and in general to provide a start-up optimisation for a single-server, you can add the following to your JVM profile.

    -Xshareclasses:name=<arbitrary_cache_name>,groupAccess,nonfatal

    As I mentioned, if none of the above helps, get the trace/log files to us and we'll take a look.

    thx

    Ivan

  • yairbar
    yairbar
    20 Posts

    Re: CWWKE0064E - Need advice: Liberty profile failure under CICS 5.1 Developer

    ‏2013-06-09T13:31:42Z  

    Hi Yair,

    The OSGI_FRAMEWORK_TIMEOUT value is not related to the timeout message you see. The controller timeout can be configured by a separate 'IBM service only' configuration setting. It is not documented because it should only be changed under IBM guidance, and in your case I don't think it is relevant or useful to change.

    Back to the failure message, this indicates that the CICS-feature of the embedded Liberty instance could not be started within a 'reasonable' amount of time. There could be so many reasons for that, and we may require a CICS trace and a dfhjvmtrc to determine the underlying problem. I suggest you set CICS trace of AP and SJ domains to ALL before you start the JVMSERVER. Then provide the CICS trace dataset, CICS joblog, and the zFS equivalents (dfhjvmtrc, dfhjvmerr, dfhjvmout) if none of the following actions help.

    I notice the value in your JVM profile:  WORK_DIR=/usr/lpp/cicsts/cicsts51/dvlp. This WORK_DIR needs to allow read/write access; while your zFS path looks like a location we would normally expect to be read only.

    I also notice you are not using the autoconfigure option. However you say you used it initially. The autoconfigure option will need to be used initially for each Liberty 'server' you start. So for example, autoconfigure of myserver1 does not mean that a name change to myserver2 will work. I say this, just in case you have changed server name without using autoconfigure for that server.

    If ensuring the WORK_DIR is read/write does not cure the problem, nor does the autoconfigure option, then it may be a genuine timeout problem. Indications of this would be:

    - you are starting many JVMSERVER Liberty servers at the same time.

    - there was a zFS i/o defect that caused very slow zFS response, I forget the details, but please ensure you have the latest service to z/OS.

    For the case where you are starting many servers simultaneously, and in general to provide a start-up optimisation for a single-server, you can add the following to your JVM profile.

    -Xshareclasses:name=<arbitrary_cache_name>,groupAccess,nonfatal

    As I mentioned, if none of the above helps, get the trace/log files to us and we'll take a look.

    thx

    Ivan

    Hello Ivan,

    Thanks for comments!

    We've found out that zfS is at highest maint.level. It also looks like there was no authorization prob.

    But the tesing LPAR suffers of a small machine resources slice while entire machine is usually  heavily loaded.

    Sometimes the Liberty JVMServer is Enabled OK, but most of times it fails with the 'Wait for Controller has timed out'' notification as appears in its dfhjvmerr log.

    We attach here all tracing & logs as you suggested - may be it will give you more info. We'ld like to make this JVM 's intialization  stable and relyable even if its LPAR is 'weak' . We dont mind it to run slowly, but please dont 'timeout' (if possible)..  

    Anyhow please note that this malfunctioning that we are experiencing  -( let its cause be any infrastructure prob or any config error   by   us) - is not diagnosed by  CICS in a comprehensive &  self axplanatory  message.

    Thanks again very much  for helping us a little further !!

    Regards,

    Yair

  • IvanHargreaves
    IvanHargreaves
    27 Posts

    Re: CWWKE0064E - Need advice: Liberty profile failure under CICS 5.1 Developer

    ‏2013-06-10T14:24:03Z  

    Hi Yair,

    I cannot see any problems with your setup. The trace shows me everything is working as expected, and it really does look like the timeout happens because of the lack of hardware resources and the constrained test LPAR. I have to admit we thought a 60 second default timeout was plenty of time in which Liberty could initialize in any useable environment. I understand your frustration with the lack of information in the failure message, but it is Liberty in control at this point while CICS sits waiting for the 'ready' message. If CICS does not get that 'ready' message in a timely fashion we cannot know whether the hardware is running slow, or whether the CICS feature was simply not installed, or whether it was installed and something failed. Hence the rather generic message.

    The good news is that we (CICS) had already requested a 'formal' notification mechanism to be added to Liberty, so that we can control the initialisation with a proper API. This API is available in v8.5.5 of Liberty. I would expect CICS to be able to take advantage of that in any future developments.

    In the meantime, there is a internal property which you can use to control the amount of time CICS will wait for Liberty to initialise. By internal I mean a property that is not documented, and would normally only be used under IBM service guidance.

    Try adding the following value to your JVM profile:

    -Dcom.ibm.cics.jvmserver.controller.timeout=120000

    The value is milliseconds. When not specified, the default is 60000.

    I hope that by increasing the timeout value, it allows you to achieve consistent (if not very slow) start-up of the Liberty server in CICS.

    thx

    Ivan