How to enable dumps in IBM z/OS Connect
Learn how to enable dumps when you are asked to do so by IBM Support.
Dumps are an important source of detailed information about problems. They give you a snapshot of what was happening in z/OS Connect at the moment the dump is taken.
z/OS Connect Server dump
Started task Applies to z/OS Connect Servers run by using a z/OS started task procedure.
- Contains
- Compressed server file. Contains trace.log, messages.log, FFDCs, resources . For example, .aar , .sar and .ara files), and server.xml. It does not contain included xml files and resources that are not in the default locations.
- Collecting
- Enter the following command.
/F <job_name>,DUMP - Location
- WLP_USER_DIR/servers/<servername>
- Filename
- <server_name>.dump-<date>_<time>.zip
More information to set up trace strings and request dumps for Liberty, see 
Set up
trace and get a full dump for WebSphere Liberty.
Javacore dumps
Started task Applies to z/OS Connect Servers run by using a z/OS started task procedure.
- Contains
- Information about the running JVM process, including environment, loaded libraries, snapshot of running threads, GC (Garbage Collection) history, storage management, and deadlocks.
- Collecting
- Automatically on some Java errors.
- Location
- Written to the location specified by the first available (allowing for existence and write
permissions) of the following.
- The location is specified by the _CEE_DMPTARG environment variable.
- The home directory of the user running the z/OS Connect Server job.
- Filename
- Default file name JAVADUMP.YYYYMMDD.HHMMSS.PID.txt (YYYY=year, MM=month, DD=day, SS=second, PID=processID).
Java Transaction dump (TDUMP)
Started task Applies to z/OS Connect Servers run by using a z/OS started task procedure.
- Contains
- An unformatted storage dump generated by z/OS.
- Collecting
- Created automatically.
- Location
- Written to a sequential MVS data set that uses a default
name of the form
%uid.JVM.TDUMP.%job.D%y%m%d.T%H%M%S, or as determined by the setting of the JAVA_DUMP_TDUMP_PATTERN environment variable. - Default Dataset Name
- %uid.JVM.TDUMP.%job.D%y%m%d.T%H%M%S.
Java heapdump
Started task Applies to z/OS Connect Servers run by using a z/OS started task procedure.
- Contains
- Portable heapdump (.phd) files contain a header, plus several records that describe objects, arrays, and classes.
- Collecting
- Created automatically when Java heap is exhausted.
- Location
- As specified by the _CEE_DMPTARG environment variable.Note: Java™ -Xdump options can be used to change the Java dumps that are produced on the various types of signals and the naming conventions for the produced dumps. The OpenJ9: Xdump Option Builder can be used to build the -Xdump options and check the available parameters.
Specifying -Xdump:what in the z/OS Connect server's JVM options cause the locations of the different Java dumps to be written to the job's STDERR on startup. For example, the following is output written as a result of specifying -Xdump:what
----------------------
-Xdump:system:
events=gpf+user+abort+traceassert+corruptcache,
dsn=%uid.JVM.%job.D%y%m%d.T%H%M%S.X&DS,
range=1..0,
priority=999,
request=serial
----------------------
-Xdump:system:
events=systhrow,
filter=java/lang/OutOfMemoryError,
dsn=%uid.JVM.%job.D%y%m%d.T%H%M%S.X&DS,
range=1..1,
priority=999,
request=exclusive+compact+prepwalk
----------------------
-Xdump:heap:
events=systhrow,
filter=java/lang/OutOfMemoryError,
file=/u/userID/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd,
range=1..4,
priority=500,
request=exclusive+compact+prepwalk,
opts=PHD
----------------------
-Xdump:java:
events=gpf+user+abort+traceassert+corruptcache,
file=/u/userID/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt,
range=1..0,
priority=400,
request=exclusive
----------------------
-Xdump:java:
events=systhrow,
filter=java/lang/OutOfMemoryError,
file=/u/userID/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt,
range=1..4,
priority=400,
request=exclusive
----------------------
-Xdump:snap:
events=gpf+abort+traceassert+corruptcache,
file=/u/userID/Snap.%Y%m%d.%H%M%S.%pid.%seq.trc,
range=1..0,
priority=300,
request=serial
----------------------
-Xdump:snap:
events=systhrow,
filter=java/lang/OutOfMemoryError,
file=/u/userID/Snap.%Y%m%d.%H%M%S.%pid.%seq.trc,
range=1..4,
priority=300,
request=serial
----------------------
-Xdump:jit:
events=gpf+abort,
file=/u/userID/jitdump.%Y%m%d.%H%M%S.%pid.%seq.dmp,
range=1..0,
priority=200,
request=serial
----------------------MVS system dump
Started task Applies to z/OS Connect Servers run by using a z/OS started task procedure.
- Contains
- Address space contents (depending on specification).
- Collecting
- Created automatically on some system errors and manually by using SDSF MODIFY command and related responses.
- Location
- MVS system dump data set (as referenced by the IEA611I message). For example,
IEA611I COMPLETE DUMP ON DUMP.MV23.#MASTER#.D190222.T123121.S00007 575 DUMPID=007 REQUESTED BY JOB (*MASTER*) FOR ASIDS(003D,0010) - For more information about the SYSMDUMP DD card that is required for dumps to be written on some
abend, see
SYSABEND, SYSMDUMP, and SYSUDUMP DD statements in the
z/OS documentation.
Collecting JVM dumps using containers
zosConnect-3.0 Applies to zosConnect-3.0.
Containers Applies to z/OS Connect container deployments.
Learn how to collect JVM dumps with containers to help diagnose problems at the JVM level with IBM z/OS Connect.
JVM dumps are useful for diagnosing problems at the JVM level, such as hung threads, deadlocks, excessive processing, excessive memory consumption, memory leaks, and defects in the virtual machine. The commands in this task must be run with the same privileges as other Docker commands on the host system. For example, as a superuser or by using sudo.
The following procedure uses Docker as an example. Other OCI compliant container platform runtimes are supported.
-
Run the Docker exec command
In this command,docker exec -it containerId /opt/ibm/wlp/bin/server javadump defaultServer --include=thread,heap,systemcontainerIdspecifies the name of the Docker container or the container ID. The command generates javacore, heapdump, core dumps in the container.For example,docker exec -it 0d090ff1dd4c /opt/ibm/wlp/bin/server javadump defaultServer --include=thread,heap,system Dumping server defaultServer. Server defaultServer dump complete in /opt/ibm/wlp/output/defaultServer/javacore.20220208.095910.1.0002.txt. Server defaultServer dump complete in /opt/ibm/wlp/output/defaultServer/heapdump.20220208.095910.1.0003.phd. Server defaultServer dump complete in /opt/ibm/wlp/output/defaultServer/core.20220208.095911.1.0004.dmp. - Copy the generated files to the host system.Run the following docker command:
In this command,docker cp containerName:fileToCopy directoryPathcontainerNamespecifies the name of the Docker container.fileToCopyspecifies the name of the generated file to copy.directoryPathspecifies the local directory to copy the file.
For example,docker cp docker_zosConnect_1:/opt/ibm/wlp/output/defaultServer/javacore.20220208.095910.1.0002.txt /logsToIBM/ docker cp docker_zosConnect_1:/opt/ibm/wlp/output/defaultServer/heapdump.20220208.095910.1.0003.phd /logsToIBM/ docker cp docker_zosConnect_1:/opt/ibm/wlp/output/defaultServer/core.20220208.095911.1.0004.dmp /logsToIBM/Note: To avoid space issues, delete the collected dumps in the container after copying to the host system.
Collecting diagnostic information for a Docker container
zosConnect-3.0 Applies to zosConnect-3.0.
Containers Applies to z/OS Connect container deployments.
Use the Docker inspect command to gather diagnostic information about the Docker containers that run z/OS Connect Designer and IBM z/OS Connect Server images.
The following procedure uses Docker as an example. Other OCI compliant container platform runtimes are supported.
- Run the following Docker inspect
command.
In this command,docker inspect containerIdcontainerIdspecifies the name or ID of the Docker container. -
The command generates detailed information about container configuration.
For example,
docker inspect 52f71f22f171 [ { "Id": "52f71f229bf7e109d0718cd523e7ac3d9a9348feb796885603a", "Created": "2022-02-04T08:57:16.7283508Z", "Path": "/opt/ibm/helpers/runtime/docker-server.sh", "Args": [ "/bin/sh", "-c", "/opt/ibm/wlp/bin/server run defaultServer --clean" ], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 86623, "ExitCode": 0, "Error": "", "StartedAt": "2022-02-04T08:57:17.6632832Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:3ecf1cce8007708f8eeb4bea618134db204d109e4832ea7ddb08303c54ad3345", "ResolvConfPath": "/var/lib/docker/containers/52f71f229bf7e109d0718cd523e7ac3d9a9348feb796885603a/resolv.conf", "HostnamePath": "/var/lib/docker/containers/52f71f229bf7e109d0718cd523e7ac3d9a9348feb796885603a/hostname", "HostsPath": "/var/lib/docker/containers/52f71f229bf7e109d0718cd523e7ac3d9a9348feb796885603a/hosts", "LogPath": "/var/lib/docker/containers/52f71f229bf7e109d0718cd523e7ac3d9a9348feb796885603a/52f71f229bf7e109d0718cd523e7ac3d9a9348feb796885603a-json.log", "Name": "/docker_zosConnect_1", "RestartCount": 0, "Driver": "overlay2", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [ "/Users/Alan/vNext_vols/config/certs:/config/resources/security:ro", "/Users/Alan/vNext_vols/logs:/logs:rw", "/Users/Alan/vNext_vols/project:/workspace/project:rw" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "docker_default", "PortBindings": { "9080/tcp": [ { "HostIp": "", "HostPort": "9080" } ], "9443/tcp": [ { "HostIp": "", "HostPort": "9443" } ] }, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": [], "CapAdd": null, "CapDrop": null, "CgroupnsMode": "host", "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": null, "GroupAdd": null, "IpcMode": "private", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": null, "DeviceCgroupRules": null, "DeviceRequests": null, "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": [ "/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ], "ReadonlyPaths": [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] }, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/fc14317dcf0cc8ef631b22115be90be09062cdcaed6ee1587aa1ac3-init/diff:/var/lib/docker/overlay2/jekfxaloddl18rbm7iaadu0py/diff:/var/lib/docker/overlay2/exbsxsfmiuti01m3tlum36dtz/diff:/var/lib/docker/overlay2/v4kvy2ptn415p4fjxw48f1obz/diff:/var/lib/docker/overlay2/qzon0e8ovdvbnkfs5d5o52w9v/diff:/var/lib/docker/overlay2/tqvrjp0636rrzjeqwjwyd9qfd/diff:/var/lib/docker/overlay2/dgezuswwjvipnigxts8xqkyk8/diff:/var/lib/docker/overlay2/trj0n6vvuglxahpxvfo6nnyzi/diff:/var/lib/docker/overlay2/0ih4cp1qdjpnqic74qip1nko7/diff:/var/lib/docker/overlay2/igw1ke244jcfq8tg4ov5wic18/diff:/var/lib/docker/overlay2/o8ul250pmy5gtv5sbwizm6xpx/diff:/var/lib/docker/overlay2/xxrec031ii5vikyei6s4kxckk/diff:/var/lib/docker/overlay2/74ecc925d83a3b2d233dc32dd943f27c9365363aa3bfe32c29a1cb05e7e2c795/diff:/var/lib/docker/overlay2/12b9bc52156a9cf4659b92fb8deed281f66cb199ab5f04ee9accf72ae0e29d6b/diff:/var/lib/docker/overlay2/37b5c2171af741823de45732398ff74080fa8791740ecb71c1a07ecd729f2499/diff:/var/lib/docker/overlay2/ce594ebe619ed00df8ed32dfed9bffc35b6cbd2086fa57a0fc70b227920e4ec6/diff:/var/lib/docker/overlay2/899732ae5f19c6bea03a067cdecad1ab93228cd9f5b7b59de5cb6059cba0e05f/diff:/var/lib/docker/overlay2/4922d1cf2df637ea3e5ea59c1158265bb5f02a14ad5cb5c87895a939121db0a1/diff:/var/lib/docker/overlay2/aaecc99e90095977e3ab1100a614a6c76b973d75fa683da49560d5f7ec79447c/diff:/var/lib/docker/overlay2/15423bc339537cd661083df24c978757f6577bde1026884825dde0cc903d32ab/diff:/var/lib/docker/overlay2/680027ebb922e93c05ec22c281f25d6e4e7db376948f6e5f39bad4a2a44418b1/diff:/var/lib/docker/overlay2/82c633ca5e6396fcca284d3e6e4691d6eaf5552bd681170d5554b1cc87216177/diff:/var/lib/docker/overlay2/271a3c9321d8a83823307d1d8a7846720f7a18b5fdeade957c448a35834c67c4/diff:/var/lib/docker/overlay2/5e8642d17dc1c67d6dacfce20aa77a76f1ac8b17ab962c43e98b6edc3f8b371b/diff:/var/lib/docker/overlay2/fb05610c0fe108fe13cfddd1c36191ff15ec6779595fe4c9dd333d4d084fed75/diff:/var/lib/docker/overlay2/ecac2ee05c661a41149f5a69494ef5a65d9f19a96578a86a6e1c95aac469f196/diff:/var/lib/docker/overlay2/23535bd92a4089e773c212b26140a4b7772046e7e1dbbfb9e98e7073d91c372b/diff:/var/lib/docker/overlay2/5f6adb6a6ff2c6b82aa53cc223b2fbe485c4e3da394b2a7aba5634e578d8c8b7/diff", "MergedDir": "/var/lib/docker/overlay2/fc14317dcf0cc8ef631b22115be90be09062cdcaed6ee1587aa1ac3/merged", "UpperDir": "/var/lib/docker/overlay2/fc14317dcf0cc8ef631b22115be90be09062cdcaed6ee1587aa1ac3/diff", "WorkDir": "/var/lib/docker/overlay2/fc14317dcf0cc8ef631b22115be90be09062cdcaed6ee1587aa1ac3/work" }, "Name": "overlay2" }, "Mounts": [ { "Type": "bind", "Source": "/Users/Alan/vNext_vols/config/certs", "Destination": "/config/resources/security", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/Users/Alan/vNext_vols/logs", "Destination": "/logs", "Mode": "rw", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/Users/Alan/vNext_vols/project", "Destination": "/workspace/project", "Mode": "rw", "RW": true, "Propagation": "rprivate" } ], "Config": { "Hostname": "52f71f22f171", "Domainname": "", "User": "1001", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "ExposedPorts": { "9080/tcp": {}, "9443/tcp": {} }, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "DB2_USERNAME=db2Username", "DB2_PASSWORD={aes}AOWLU52IW8jKyHwGCXqjgRfE9m9bj3sxc8FmuAr/biA1FpINk/sktiZWrBZQ+NFgZQ==", "PATH=/opt/gradle/bin:/opt/ibm/java/bin:/opt/ibm/wlp/bin:/opt/ibm/helpers/build:/opt/ibm/java/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "container=oci", "JAVA_VERSION=8.0.6.36", "JAVA_HOME=/opt/ibm/java/jre", "IBM_JAVA_OPTIONS=-XX:+UseContainerSupport", "LIBERTY_VERSION=21.0.0_09", "LOG_DIR=/logs", "WLP_OUTPUT_DIR=/opt/ibm/wlp/output", "OPENJ9_SCC=true", "RANDFILE=/tmp/.rnd", "WLP_LOGGING_CONSOLE_LOGLEVEL=info", "WLP_LOGGING_CONSOLE_SOURCE=message,trace,accessLog,ffdc,audit", "GRADLE_HOME=/opt/gradle", "GRADLE_USER_HOME=/gradle" ], "Cmd": [ "/bin/sh", "-c", "/opt/ibm/wlp/bin/server run defaultServer --clean" ], "Image": "ibm-zcon-designer:latest", "Volumes": { "/config/resources/security": {}, "/logs": {}, "/workspace/project": {} }, "WorkingDir": "", "Entrypoint": [ "/opt/ibm/helpers/runtime/docker-server.sh" ], "OnBuild": null, "Labels": { "BuildLabel": "cl210920210824-2341", "ProductID": "fbf6a96d49214c0abc6a3bc5da6e48cd", "ProductName": "WebSphere Application Server Liberty", "ProductVersion": "21.0.0.9", "architecture": "x86_64", "build-date": "2021-09-14T16:37:54.438551", "com.docker.compose.config-hash": "5e99f9e2bf6cc9cf4dd5a79ad6f84a36acf6eced52823c41b", "com.docker.compose.container-number": "1", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "docker", "com.docker.compose.project.config_files": "docker-compose.yaml", "com.docker.compose.project.working_dir": "/Users/Alan/vNext/scripts/docker", "com.docker.compose.service": "zosConnect", "com.docker.compose.version": "1.29.2", "com.redhat.build-host": "cpt-1002.osbs.prod.upshift.rdu2.redhat.com", "com.redhat.component": "ubi8-container", "com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI", "description": "This image contains the WebSphere Liberty runtime with IBM's Java and Red Hat's UBI 8 as the base OS. For more information on this image please see https://github.com/WASdev/ci.docker#building-an-application-image", "distribution-scope": "public", "io.k8s.description": "The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.", "io.k8s.display-name": "Red Hat Universal Base Image 8", "io.openshift.expose-services": "", "io.openshift.tags": "base rhel8", "maintainer": "Red Hat, Inc.", "name": "IBM WebSphere Liberty", "org.opencontainers.image.authors": "Arthur De Magalhaes, Chris Potter", "org.opencontainers.image.documentation": "https://www.ibm.com/support/knowledgecenter/SSAW57_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/cwlp_about.html", "org.opencontainers.image.revision": "cl210920210824-2341", "org.opencontainers.image.url": "http://wasdev.net", "org.opencontainers.image.vendor": "IBM", "org.opencontainers.image.version": "21.0.0.9", "release": "8", "run": "docker run --rm -ti \u003cimage_name:tag\u003e /bin/bash", "summary": "Image for WebSphere Liberty with IBM's Java and Red Hat's UBI 8", "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.4-211", "vcs-ref": "ed5adf70c28eb951940c7a32c4bca2165", "vcs-type": "git", "vendor": "IBM", "version": "21.0.0.9" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "fb80e2a2eae852ee6ee9833644886a64fa9e9e5cca2559", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": { "9080/tcp": [ { "HostIp": "0.0.0.0", "HostPort": "9080" }, { "HostIp": "::", "HostPort": "9080" } ], "9443/tcp": [ { "HostIp": "0.0.0.0", "HostPort": "9443" }, { "HostIp": "::", "HostPort": "9443" } ] }, "SandboxKey": "/var/run/docker/netns/fb80e2a2eae8", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "docker_default": { "IPAMConfig": null, "Links": null, "Aliases": [ "52f71f22f171", "zosConnect" ], "NetworkID": "f7a0ab1466ce1da8d3aebe46716e74eb5013e3056ab51809bf15739f16c", "EndpointID": "2e4309db5e028601bf23bbdb43f70053a6180080182549234b159d7817", "Gateway": "192.168.0.1", "IPAddress": "192.168.0.2", "IPPrefixLen": 20, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "02:99:c9:b8:00:02", "DriverOpts": null } } } } ]