Pinned topic JVMPROFILE_ERROR

Hello,

Sorry to hear you have had an issue starting your JVMServer. One thing to check in the DFHOSGI JVMProfile is that JAVA_HOME is set correctly and pointing to Java 7 (64bit). If it is setup correctly then have a look in SYSPRINT of your CICS job output this will help to identify the reason the JVMSERVER is not enabling correctly. If you are not getting any output in SYSPRINT you will need to enable trace for the SJ domain (either using CETR or STNTRSJ=ALL) and attempt to enable the JVMSERVER again.

Cheers

Adam Coulthard

When I got this error, Java home was pointing to Dir J6.0.1_64. I now have changed it to Dir J7.0_64. I am getting the same error message. The addition error message that I am getting is this:
BROWSE /CICSIVP1.JVMSERV1.dfhjvmerr Line 00000000 Col 001 04
Command ===> Scroll ===> PAG

• Top of Data ********************************

Error: Port Library failed to initialize: -1

and
BROWSE /CICSIVP1.JVMSERV1.dfhjvmtrc Line 00000000 Col 001 078

• Top of Data **********************************

2012/10/31 12:48:43.901430 GMT Exc: Create JVM failed
2012/10/31 12:48:43.901641 GMT Exc: Unable to create JVM
2012/10/31 12:48:43.901690 GMT Exc: Return code is -1 meaning: Unknown error

Thanks.

Hi,

can you confirm that the USSHOME SIT parameter is set to the correct ZFS, i.e. from the same install as the CICS libraries? If it is, then as Adam said, please set SJ and AP trace to ALL, enable your JVM server and post the full contents of SYSPRINT, dfhjvmerr and dfhjvmtrc.

Thanks
Dave

Hello,

I've just been looking around for that error message and I've hit on technote that might help:

http://www-01.ibm.com/support/docview.wss?uid=swg21418742

Cheers

Adam Coulthard

Hello:
My usshome is defined in the startup JCL as an override and is not part of the sit parameter.
I changed the region size from 1150M which didnt work for me to 0M and the memlim from 8G to 12G. I turned on tracing and included the output below. The tracing output is truncated to the right. I looked at the item from Techdocs and it sounds like my problem, but I have not used IPCS in ages.
Regards,
Dietmar Demeter
//CICS EXEC DFHSTART,
// INDEX1='CICSTS51', HIGH-LEVEL INDEX FOR CICS DATASETS
// INDEX2='CICSTS51.CICS', HIGH-LEVEL INDEX FOR CICS LIBRARIES
// REGNAM='1', REGION NAME FOR SINGLE OR MRO REGION
// REG=0M, SIZE OF REGION @L2C
//*REG=1150M, SIZE OF REGION @L2C
// MEMLIM=12G, MEMORY LIMIT, @D51430C
//*MEMLIM=8G, MEMORY LIMIT, @D51430C
// START=INITIAL, TYPE OF CICS START-UP
// RUNCICS=YES, START CICS. YES OR NO
// DUMPTR=NO, ANALYZE DUMP/TRACE. YES OR NO
// OUTC='*', SYSOUT CLASS
// SIP=1, SUFFIX OF DFHSIP MEMBER IN SYSIN DATASET
// CICSSVC=216, SVC NUMBER FOR CICS
// USSHOME='/usr/lpp/cicsts/cics51'

- Date: 01 Nov 2012 Time: 08:22:23 GMT PID: 0033554531
Entry: ipt_control_loop
Entry: start_jvm
Event: Printing 1 setup classes
Event: Setup class = com/ibm/cics/router/Router
Event: PRINT_CLASS: Arg 0 = /usr/lpp/cicsts/cics51/lib/org.eclipse.osgi.service
Event: PRINT_CLASS: Arg 1 =
Event: PRINT_CLASS: Arg 2 = /usr/lpp/cicsts/cics51/lib/com.ibm.cics.ras.jar,/us
Event: PRINT_CLASS: Arg 3 =
Event: PRINT_CLASS: Arg 4 = /usr/lpp/db2a10/jdbc/classes/db2jcc4.jar,/usr/lpp/d
Event: PRINT_CLASS: Arg 5 =
Event: PRINT_CLASS: Arg 6 = 2
Event: Printing 1 termination classes
Event: Termination class = com/ibm/cics/router/Router
Event: LIBPATH = /usr/lpp/cicsts/cics51/lib:/usr/lpp/java/J7.0_64/bin/j9vm:/usr
Event: CWD = /
Event: OSGI_FRAMEWORK_TIMEOUT = 60
Exc: Create JVM failed
Exc: Unable to create JVM
Exc: Return code is -1 meaning: Unknown error

• Bottom of Data ********************************

Hi,

Did any files suffixed dfhwlperr, dfhwlpout or dfhwlptrc get created on ZFS in WORK_DIR as specified in the JVM profile? Looking at your output, this will be the root directory (Event: CWD = /). If you are using the default of DIR=WORK. then it looks like the region id does not have a unix home directory set. I suspect that you the your root is either read only or the ID doesn't have access to create files in the root.

So you either need to set the home for the CICS user or set WORK_DIR to a valid path

Regards
Dave

Hello:

I am trying to activate the JVM profile DFHOSGI. I did not set the working directory to a specific directory, therefore it defaults to /. The files that you mentioned are created:

File CICSIVP1.JVMSERV1.dfhjvmerr
File CICSIVP1.JVMSERV1.dfhjvmin
File CICSIVP1.JVMSERV1.dfhjvmout
File CICSIVP1.JVMSERV1.dfhjvmtrc

The contents of dfhjvmerr are:
Error: Port Library failed to initialize: -1

and the contents file dfhjmtrc are:
Entry: ipt_control_loop
Entry: start_jvm
Event: Printing 1 setup classes
Event: Setup class = com/ibm/cics/router/Router
Event: PRINT_CLASS: Arg 0 = /usr/lpp/cicsts/cics51/lib/org.eclipse.osgi.service
Event: PRINT_CLASS: Arg 1 =
Event: PRINT_CLASS: Arg 2 = /usr/lpp/cicsts/cics51/lib/com.ibm.cics.ras.jar,/us
Event: PRINT_CLASS: Arg 3 =
Event: PRINT_CLASS: Arg 4 = /usr/lpp/db2a10/jdbc/classes/db2jcc4.jar,/usr/lpp/d
Event: PRINT_CLASS: Arg 5 =
Event: PRINT_CLASS: Arg 6 = 2
Event: Printing 1 termination classes
Event: Termination class = com/ibm/cics/router/Router
Event: LIBPATH = /usr/lpp/cicsts/cics51/lib:/usr/lpp/java/J7.0_64/bin/j9vm:/usr
Event: CWD = /
Event: OSGI_FRAMEWORK_TIMEOUT = 60
Create JVM failed
Unable to create JVM
Return code is -1 meaning: Unknown error

The copy and paste is truncated to the right.

Regards,
Dietmar Demeter

A bit more info about this:

I am executing the CICS IVP job with the standard SIT parameters pointing to (DFH$SIP1).
Do I need to assemble another sit?

These are the parameters:
XRF=NO,
AUXTR=ON,
AUXTRSW=NEXT,
APPLID=CICSIVP1,
DSALIM=5M,
EDSALIM=800M,
FCT=NO,
JVMPROFILEDIR=/u/cics/JVMProfiles,
TCT=NO,
SRT=NO,
SEC=NO,
TCPIP=YES,
TRTABSZ=10000,
TSMAINLIMIT=64M,
USSHOME=/usr/lpp/cicsts/cics51,
PGRET=P/,
PGPURGE=T/,
PGCOPY=C/,
PGCHAIN=X/,
.END

Dietmar Demeter

Hi,

The parameters in the SIT look to be ok, looking at the technote it says to reduce your DSALIM size but since its already at 5MB I doubt making it smaller would help. Do you have any PLTPI programs setup to execute when CICS starts? I know this sounds odd but we have seen in the past PLT programs taking up storage at startup that stops the JVMs starting.

To progress this further (since the PLTPI removal is just a work around) we probably need to look at a dump to analyse the storage. Would you be able to take a dump and ftp it for us to look at using the following:

ftp.hursley.ibm.com

login as anonymous

p/w = internet email ID

cd to directory /pub/import/cicsbeta/openbeta

Then do a binary PUT of your tersed dataset

Once this is complete, can you please reply with the name of the dump/s you have sent in.

Would you also be able to upload your JVMProfile in text format?

Thanks

Adam Coulthard

Hello:

What dump options should I specify?

Regards,

Dietmar Demeter

• VentyxSyspro 2000002E6E
• ‏2012-11-02T16:10:49Z

Hello:

What dump options should I specify?

Regards,

Dietmar Demeter

More...

Hi,

Can you make sure that SJ=ALL is set and the internal trace table is set to a good size and take the dump after attempting an enable of the JVMServer

Thanks

Adam

230- ************************************************************************
230- * IBM's systems must only be used for conducting IBM's business *
230- * or for purposes authorized by IBM management *
230- ************************************************************************
230-
230- No Confidential material of any kind must be placed on this server.
230-
230- Any Confidential data that is discovered will be removed without warning
230-
230 Login successful.
ftp> cd /pub/import/cicsbeta/openbeta
250 Directory successfully changed.
ftp> bin
200 Switching to Binary mode.
ftp> put 'sys1.zosd.dmp00005.tersed'
200 PORT command successful. Consider using PASV.
553 Could not create file.
ftp>

This is what I am getting from your server when I try to send a tersed dump.

Regards,
Dietmar Demeter

Hello:

Its Monday morning and I tried again to FTP to your suggested site and this is what I am getting:

200 Port Command successful. Consider using PASV

553 Could not create file.

425 Failed to establish connection.

Regards,
Dietmar Demeter

• VentyxSyspro 2000002E6E
• ‏2012-11-05T13:57:21Z

Hello:

Its Monday morning and I tried again to FTP to your suggested site and this is what I am getting:

200 Port Command successful. Consider using PASV

553 Could not create file.

425 Failed to establish connection.

Regards,
Dietmar Demeter

More...

Hi Dietmar,
I'm not sure what's causing your problem. I can see that the tersed dump you uploaded has created a file on the server, but the file size is zero bytes. I did a test upload into that directory earlier and it worked okay! I'm trying to find out what's going wrong and will get back to you shortly.
Cheers, David

• davidknibb 270000DY3K
• ‏2012-11-05T14:10:39Z

Hi Dietmar,
I'm not sure what's causing your problem. I can see that the tersed dump you uploaded has created a file on the server, but the file size is zero bytes. I did a test upload into that directory earlier and it worked okay! I'm trying to find out what's going wrong and will get back to you shortly.
Cheers, David

More...

Hi Deitmar,
I think that the second test you did yesterday may have failed since the file already existed on the server from your earlier attempt (albeit with the zero size). The directory has now been cleared out, the server restarted, and I've been able to run a test upload to that directory. So at this point it appears to be working.

Could you have another go? If it doesn't work now we may have to try changing some settings at your end (e.g. try a different FTP client, use PASV instead of active etc). If you do have to retry then please retry with a different file name to avoid conflicting names.

Hopefully it should be okay now!
Thanks, David

Hello:

I tried again to send the dump to your site. I used my work PC and then my personal PC. In both instances, i am getting the same message:

Failed to establish communication. This message appears after signing in successfully to your FTP site.

I am recruiting my own IT internal Help Desk for assistance.

Any more thoughts?

Dietmar Demeter

Hello:

How about this: Can I attach the tersed file(about 4.49MB) to this thread?

Dietmar

Hello:

I have attached the tersed dump to this message.

Regards,

Dietmar

• VentyxSyspro 2000002E6E
• ‏2012-11-06T21:39:58Z

Hello:

I have attached the tersed dump to this message.

Regards,

Dietmar

More...

Hi Dietmar,
Reading your messages above, my first though was that 4.49MB does not sounds large enough for a CICS dump. I was able to pull your file down to my workstation, and upload it again to a MVS system here. I've now got a dump on MVS which can be loaded in IPCS. Unfortunately it would appear that large parts of the CICS address space are missing:
00002000.:5A0FFF. LENGTH(X'59F000')--Storage not available

One of the things in this missing area is the Kernel anchor table, so the dump cannot be formatted by the CICS dump formatters. I can see enough to see that this is a MVS Console dump of address space IBMUSERO, but we can't tell much about the storage allocations which is what we need in this case. I'm not sure what would cause this - maybe the dump data sets or the target data set on the terse step were too small?

Please could you take another dump (a dump taken from CEMT P SNAP would be fine), terse it, and try the FTP upload again?

Thanks, David

Hello:
I got the dump. 68MB. Problem I cant FTP to your site.

Getting the same FTP failure: Error code 425 Failed to Establish communication.

So now what. Can we attempt to send this dump to your normal FTP PMR site?
or shall I put it on a CD and FEDEX to an address that you could provide?

I dont think that I can attach this dump to this thread.

Regards,

Dietmar Demeter

Hello:

I took my PC to the local Starbucks to attempt the FTP to your site. Recently I installed a new LINKSYS router at my home office and I was thinking the new router could be the problem. Well, at Starbucks I am getting the following message from FTP: connection closed by remote host. I get this message after I executing the put command.

Dietmar

• VentyxSyspro 2000002E6E
• ‏2012-11-07T15:43:09Z

Hello:

I took my PC to the local Starbucks to attempt the FTP to your site. Recently I installed a new LINKSYS router at my home office and I was thinking the new router could be the problem. Well, at Starbucks I am getting the following message from FTP: connection closed by remote host. I get this message after I executing the put command.

Dietmar

More...

Hi Dietmar,
68MB sounds like a much better dump. Sorry this is proving so difficult to upload. I've just done a couple of test uploads and they appeared to complete okay so I really don't know what's going on with the FTP server. I am talking to our support team to try to work this out.
I can see that you attempted to upload SYS1.ZOSD.DMP00006.TERSED at 07-Nov-2012 14:49, but once again the file has a zero size. I can only see one file there though and it sounds like you've had a couple of goes it it. Make sure you always use a different name since I think the upload will always fail if the file is already present.
I'll get back to you later on and let you know what to do next.
Cheers, David

• davidknibb 270000DY3K
• ‏2012-11-07T15:55:12Z

Hi Dietmar,
68MB sounds like a much better dump. Sorry this is proving so difficult to upload. I've just done a couple of test uploads and they appeared to complete okay so I really don't know what's going on with the FTP server. I am talking to our support team to try to work this out.
I can see that you attempted to upload SYS1.ZOSD.DMP00006.TERSED at 07-Nov-2012 14:49, but once again the file has a zero size. I can only see one file there though and it sounds like you've had a couple of goes it it. Make sure you always use a different name since I think the upload will always fail if the file is already present.
I'll get back to you later on and let you know what to do next.
Cheers, David

More...

Hi Dietmar - please could you email me directly (davidknibbatuk.ibm.com) and we'll discuss how to proceed.
Thanks, David

Hello:

The solution to this problem has been found. The Java level of my ZOS 1.13 system was GA. The minimum requirement for Java in CICS TS 5.1 is Java 7.0 64 bit SR1 which is documented in the prerequisites. There is a URL available to download the latest Java level: http://www-03.ibm.com/systems/z/os/zos/tools/java/products/sdk7_64.html.

Thanks for all the help from IBM on this topic.

Regards.

Dietmar Demeter