IBM Support

MustGather: Collect Troubleshooting Data for FTP for the z/OS Communications Server

Troubleshooting


Problem

This technical document discusses the most common problems, configuration issues, and recommended maintenance for the FTP application on the z/OS operating system. Additionally, it discusses the methods by which diagnostic information can be obtained to troubleshoot and resolve FTP problems. This document replaces informational APARs II12079, II12925, and II13516.

Resolving The Problem


Table of Contents
General Information Recommendations Common Problems and Questions Gathering Diagnostic Output Glossary of Acronyms




General Information
Implicit TLS connections over port 990

  • The use of port 990 to implicitly protect FTP sessions was included in the early drafts of the IETF documents that describe how to use TLS with FTP, but has been removed from the latestdrafts. We strongly recommend not starting the FTP server on port 990 - if a server is started on this port, not all secure FTP clients will be able to connect to it. The FTP server can provide equivalent support on a different port by specifying the following in FTP.DATA:

      •  EXTENSIONS AUTH_TLS
       SECURE_FTP REQUIRED
       SECURE_CTRLCONN PRIVATE
       SECURE_DATACONN PRIVATE

    We also recommend not using our FTP client port 990 to connect to the FTP server. The FTP client might not be able to connect to all servers on this port. The FTP client can be configured to require TLS security by invoking the FTP client with the -r TLS option. Common Errors using port 990 for a TLS FTP based connection:
  • FTP Server trace, option SEC, might show:

      •  authClient: init failed with rc = 410 (GSK_ERR_BAD_MESSAGE)
       EZYFT96I TLS handshake failed

    or

       authClient: init failed with rc = 406 (GSK_ERR_IO)
       EZYFT96I TLS handshake failed

  • The FTP client trace, option SEC, might show:

      •  authServer: secure_socket_init failed with rc = 410
       (GSK_ERR_BAD_MESSAGE) EZA2897I Authentication negotiation failed

    or

       authServer: secure_socket_init failed with rc = 406 (GSK_ERR_IO)
       EZA2897I Authentication negotiation failed

TLS Connections through Firewalls

    TLS connections through firewalls can pose multiple challenges:
    • NAT Firewall on Client side:
        • Firewalls doing Network Address Translation (NAT) might alter the parms on FTP commands being sent, like the IP address on the PORT command. These firewalls require access to the data, which is not possible when the data has been encrypted. We recommend using Firewall Friendly data connections for the z/OS FTP client, by coding FWFRIENDLY TRUE in FTP.DATA. The z/OS FTP client will send the PASV command instead of the PORT command.
    • NAT Firewall on Server side:
        • If a NAT firewall is running on the server side, the PORT command can be used to open data connections, but the PASV command might not work. The NAT firewall will need to alter the response to the PASV command, which is not possible when the data has been encrypted. Ensure that FWFRIENDLY is set to FALSE in the client's FTP.DATA..
    • NAT Firewall on both Client and Server:
        • The EPSV command must be used by the client and understood by the server.
    • Firewalls which place requirements on data being sent on the control connection:
        • TLS encrypted FTP sessions might not work through firewalls which require data on the control connection to follow specific formats. Because the data is encrypted, rules cannot be applied to the format of the data being sent. If a firewall requires each packet end in a Carriage Return and Line Feed, the TLS encrypted packets might be rejected and the FTP session might be terminated abnormally. The firewall might need to be configured to allow TLS FTP traffic.


Recommendations
Naming the FTP Server

    It is recommended the FTP server started procedure be named 'FTPD' or something similar with LESS THAN EIGHT CHARACTER NAME. When the server is started, it will spawn a listening daemon with the proc name and a '1' appended to the end (that is, FTPD1) **
Using SYSLOGD

    SYSLOGD should be running and properly configured to capture FTP server trace records. See our SYSLOGD MustGather.

    NOTE: If syslogd is not running, all FTPD trace output will typically go to the MVS console. This is NOT recommended because the console will get flooded.
Using the Resolver

    FTP is a UNIX application, so the resolver follows the UNIX search order. See the z/OS Resolver Read First if FTP is failing to resolve hostnames.
Applying Maintenance

    When applying maintenance to the FTP client or server, both the load modules and the aliases need to be copied over. For the client, FTP and EZAFTPLC need to be copied over. For the FTP server, FTPDNS and EZAFTPLS must be copied over. And for the daemon, FTPD and EZAFTPLD. If not performed properly, unpredictable results might occur, including abends.
Userids and Passwords
    When invoking FTP via shell script or MVS batch job, and encountering "Permission Denied" message such as

      530 PASS command failed - __passwd() error

    ensure that userid/password are correctly authorized, and that the word PASS is not included as part of the command. For example, use the following syntax:

      USER userid password

    === OR ===

      USER userid
      password

    Note: Do not use the PASS subcommand here.


Common Problems and Questions
Symptoms:
  • 530 Pass Command Failed
  • 530 - A load was done from an uncontrolled library
  • 530 Pass command failed: passwd() failed
  • EDC5157I Internal error
  • ICH420I messages indicating a program was loaded causing the environment to be uncontrolled.


Resolution: Any of the following might be causes of the above problems:
  • The sticky bit must be turned on for both /usr/sbin/ftpd and /usr/sbin/ftpdns. You can verify that the sticky bit is on by executing the following command:

      •  ls -l /usr/lpp/tcpip/sbin/ftp*

    The output will look similar to: 

       -rwxr-xr-t  17 OMVS ...  Jan 27 1998 /usr/lpp/tcpip/sbin/ftpd   -rwxr-xr-t  17 OMVS ...  Jan 27 1998 /usr/lpp/tcpip/sbin/ftpdns           / \       STICKY BIT

    The "t" in the permission bits indicates that the sticky bit is set. Use the follow command to set the sticky bit if the "t" is not present: 

       chmod o+t /usr/lpp/tcpip/sbin/ftp*
  • Copies of ftpd, ftpdns, and the LE libraries (such as SCEERUN) must reside in authorized data set(s) in the linklist.
  • In order for the FTPD cataloged procedure to get control with superuser and daemon authority, you must add an entry to the started procedures tables in RACF (ichrin03):

      •  DC  cl8'ftpd'  procedure name   DC  cl8'ftpd'  procedure name   DC  cl8'ftpd'  userid   DC  cl8'    '   DC  xl1'40'    trusted user   DC  xl7'00'    reserved
  • Authorize userids to the system with an OMVS segment (i.e OMVS default segment)
  • Directories in the path to the FTP executable must be set to 755.
  • Ensure that you are running the libraries that came with your release, that is, LE/370 or C runtime.
  • Ensure that the following libraries are defined as program controlled:
    • C/C++ run-time libraries
    • Language Environment libraries
    • SYS1.LINKLIB
    • SYS1.SIEALNKE
  • ACCESSERRORMSGS TRUE or DEBUG ACC can be coded in FTP.DATA to generate more error information for failed PASS commands.
Symptom: GDG transfers fail

Resolution: A model DCBDSN must exist in the FTP.DATA file as DCBDSN=model. To use a DCBDSN model to create a data set, do the following:
  1. Issue the following command:

    SITE DCBDSN=data_set_name

    where data_set_name is the name of the data set to be used as a model to set the values of the logical record length (LRecl), block size (BLKsize), retention period (RETpd), and the record format (RECfm) of a new data set.

  2. Issue the following command to enable the LRecl, BLKSIze and RECfm of the model to be used:

  3. Issue the following command to create the new data set with the values specified by the DCBDSN model:

    PUT data_set_name

    where data_set_name is the name of the new data set.
Symptom: 530 PASS command failed - getpwnam() error : USERNAME

Resolution: Set up a superuser FTPD userid. Ensure you have defined a HOME directory ( / ) and then add the userid to RACF using the command: 

    ADDUSER FTPD OMVS(UID(0) HOME('/') PROGRAM('/bin/sh'))

Permit it to BPX.DAEMON facility if necessary: 

    PERMIT BPX.DAEMON CLASS(FACILITY) ID(FTPD) ACCESS(READ)
Symptom: FTP listens on multiple stacks when single stack affinity is desired.

Resolution: Use ENVAR to ensure the server binds to the correct stack. For example: 

      //FTPD   PROC MODULE='FTPD',PARMS='TRACE'  //FTPD   EXEC PGM=&MODULE,REGION=7M,TIME=NOLIMIT,  //       PARM=('POSIX(ON) ALL31(ON)',  //      'ENVAR("_BPXK_SETIBMOPT_TRANSPORT=xxxxxxxx")',  //      '/&PARMS')
      //* where 'xxxxxxxx' is the stack name for affinity
Symptom: The following error is generated: 
    ICH408I USER(aaaaaa) GROUP(bbb) NAME(ccccc)  /usr/sbin/ftpdns  CL(DIRSRCH) FID(dddddddddd)  INSUFFICIENT AUTHORITY TO LOOKUP  ACCESS INTENT(---X) ACCESS ALLOWED(GROUP ---)

Resolution: The key being CL(DIRSRCH) - meaning that RACF was trying to do a directory search and the user was not allowed. Either root ( / ) or one of the subdirectories does not have its permission bits 755. Verify with the command 

    ls -ld /

(for root). The output should look like: 

    drwxr-xr-x ....

This should be the setting for each subdirectory as well as root. Issue the command 

    chmod 755 /

(for root) to add the correct permission bits.
Symptom: Receiving EZA2562W with reason code 536 when attempting to transfer/allocate a file to a tape data set.

Resolution: Make sure AUTOTAPEMOUNT=TRUE is specified in the CLIENT FTP.DATA file.
Symptom: Top Secret or ACF2 users applying PQ63326.

Resolution: A new resource profile must be defined to the SERVAUTH facility class to allow users to access the HFS (EZB.FTP.sysname.ftpdaemonname.ACCESS.HFS)
Symptom: FTPD fails on startup with 
    EZYFT12E socket error : EDC5111I Permission denied.
or 
    EZYFT13E bind error : EDC5111I Permission denied.

Resolution: This is caused by having SERVAUTH active and ACF/2 not having EZB.STACKACCESS.sysname.tcpname defined. The ACF2 R10 compatibility fixes add the SERVAUTH Class to the Class Profiles that respond to SAF Calls as being active. So once the R10 compatibility service is installed on ACF2 6.3 (the only release supported on R10), the SERVAUTH Class is active (as far as TCP/IP is concerned) and the Installation MUST create the various EZB.xxxxx Profiles in the SER Resource Type (which ACF2 maps SERVAUTH to). The existence of the SERVAUTH Class mapping would have no impact on previous releases, as no one was using them.
Symptom: FTP Server Hangs

Resolution: This problem is non-recoverable. It is suggested that you perform the normal termination procedures as per your operating environment (that is, purge,cancel,force). The documentation needed to identify the problem is a dump of the ftpd server, tcpip, and omvs address space along with the dataspace for omvs. The syntax for the dump command is: 

    DUMP COMM=(FTP Server Hung)  R XX,JOBNAME=(tcpprocname,ftpprocname1,OMVS),DSPNAME=('OMVS'.*),  SDATA=(CSA,RGN,TRT,SUM,ALLNUC),END

Here is an example for the Reply XX (above assuming the tcprocname is TCPIP and the ftpprocname is FTPD): 

    R 00,JOBNAME=(TCPIP,FTPD1,OMVS),DSPNAME=('OMVS'.*),  SDATA=(CSA,RGN,TRT,SUM,ALLNUC),END
Symptom: FTP Fails when BPX.POE class is active

Resolution: This is caused by not having the proper FTP definitions when the SERVAUTH CLASS is activated. When activated:
  • NETACCESS profile must be defined for each network security zone.
  • Authorize FTPD to NETACCESS profiles for read access from which any client can log in.
  • Authorize FTP login users to NETACCESS profiles for read access from which they can log in.
  • Add a PORTOFENTRY4 SERVAUTH statement to FTP.DATA if IPv4 clients are to be migrated to SERVAUTH
  • Define data sets which are to have limited access by Port of Entry.
See z/OS Communications Server IP Configuration Guide for additional information.
Symptom: FTP Client fails with message EZA2897I Authentication negotiation failed.

Resolution: Obtain a FTP client trace with the SEC CMD SOC(3) and FLO options. If trace shows:
  • ftpAuth: TLS init failed with rc = 408 (GSK_ERR_BAD_KEYFILE_PASSWORD):
      • This can occur when using gskkyman. Ensure that the password is stored in a stash file and that the FTP client can access this file.
  • authClient: init failed with rc = 410 (GSK_ERR_BAD_MESSAGE):
      • No keyring file was specified in FTP.DATA.
  • authClient: 068 init failed with rc = 422 (GSK_ERR_BAD_V3_CIPHER):
      • If using CIPHERSUITE keyword in FTP.DATA, ensure the proper System SSL FMIDs have been installed for the ciphers specified and the CIPHERSUITE statements need to be coded in the order of preferred usage.


Gathering Diagnostic Output
General Guidelines for Collecting Output

  1. Identify if this an FTP client or FTP server related problem.
  2. Always provide a copy of any FTP client output since the FTP session was started.
  3. If any error message is prefixed by 3 digits, such as 
    550 Data set SYS1.TCPPARMS not found
    this is an error being reported by the FTP server. Provide documentation as documented for server problems.
  4. Provide a copy of the FTP.DATA file for the failing applicaton (client or server).

    NOTE: As an alternative, use following FTP commands from the FTP client 
    QUOTE STAT    ; for z/OS FTP server variables 
    LOCSTAT       ; if client is a z/OS Client for z/OS FTP client variables
  5. Review console log around time of error and supply console output for any FTP messages
    NOTE: Small text output can be emailed to mvs_support@ecurep.ibm.com
    The subject line should be include
    PMR [pmr_number],[branch code],[country code]
    as otherwise the PMR will not be updated indicating an email has been received.
  6. Provide a packet trace if
    • The problem being reported concerns a timeout, reset, or performance
    • The problem being reported involves FTP commands being sent to a z/OS FTP server via a browser. See How to Collect PKTTRACE and CTRACE on z/OS for instructions on obtaining a packet trace.

Common DEBUG options
Type of problem
Options to set
SSL/TLS ProblemsSOC(3),SEC,CMD,FLO
Logon ProblemsACC,CMD,FLO
Timeout ProblemsFLO,FSC(3),CMD
Transfer ProblemsINT,FSC(3),UTL,FLO,SOC(3)
JES ProblemsJES,FLO,FSC(3)


Common DUMP options
Type of problem
Options to set
File Access ProblemsFSC
Network ProblemsSOC
  • For the Data connection, set level 52
  • For the Control connection, set level 54: SOC(54)
JES ProblemsJES
SQL ProblemsSQL
Translation Table Problems81
User Exit Problems82 and 87
Socks84 and 85
Documentation needed for FTP Server problems

USER abends (ie, U40xx)
For user abends, check for an existing CEEDUMP on your system. Information on CEEDUMPs can be found in the Language Environment Programming Reference. If the Level 2 support team decides the CEEDUMP is insufficient, an SVC dump trapping the original abend might be required. In this case, the following actions should be taken:
  1. Code the following in the PARM specification of the FTPD cataloged procedure:
     TRAP(ON,NOSPIE) TERMTHDACT(UAIMM)
  2. Set a SLIP to trap the original abend. Typically, the original abend code can be found in message IEF450I. If unable to determine abend code, contact level2 support.

Gathering FTP server traces
Tace output will go to a file in the HFS as defined by the syslogd configuration file (/etc/syslog.conf). For additional information regarding syslogd, see the Syslogd MustGather / Read First.

NOTE: If syslogd is not configured for FTP on your system, the trace output will be written to the console. This is *NOT* recommended and can result in the console being flooded. Output that is sent to the console is also difficult to read and might delay problem diagnosis.

Code the following statement in the Server's FTP.DATA file to allow FTP clients to selectively activate traces. Recycle the server to activate this support. 

 DEBUGONSITE TRUE

Different methods of activating server traces
Using Modify commands to set DUMP or DEBUG options
The following modify command can be used to activate FTP server traces
 MODIFY ftp_server_jobname,DEBUG=(xxx)

where xxx can be multiple server trace options as shown above. To disable the tracing the command can be issued as follows:
 MODIFY ftp_server_jobname,DEBUG=(NONE)

To display the current settings, issue
 MODIFY ftp_server_jobname,DEBUG=(?)

Setting trace/debug options in the FTP.DATA configuration file
Debugging can also be activated by adding configuration statements to the FTP.DATA configuration file. The DEBUG statement can be coded for each option desired. For example, to turn on BAS, FLO, and ACC, code
 DEBUG BAS   DEBUG FLO   DEBUG ACC

The TRACE statement can also be coded:
 TRACE

This is the equivalent of coding
 DEBUG BAS

Using the FTP client SITE command to activate DUMP and DEBUG options on the FTP server
If DEBUGONSITE TRUE has been configured for an FTP server, the FTP client can be used to activate tracing on that server. The subcommands for turning on a specific DUMP or DEBUG option are:
 SITE DEBUG=(option1,option2,...)   SITE DUMP=(option1,option2,...)

To display a lis of options,
 SITE DEBUG=?   SITE DUMP=?

To deactivate DUMP or DEBUG tracing on the server,
 SITE DEBUG=NONE   SITE DUMP=NONE

Note that if a client other than the MVS FTP client is used, these commands must be preceded with the QUOTE keyword:
 QUOTE SITE DEBUG=(option1,option2,...)   QUOTE SITE DUMP=(option1,option2,...)
Documentation needed for FTP Client problems

Gathering FTP client traces
If the FTP session is running interactively (in other words, if a user is sitting at a terminal, entering commands in response to the FTP prompts), the trace output will be written to the user's screen. Capturing the trace will require cutting and pasting the output into a file as it is written to the screen.

If, however, the FTP session is running in batch (in other words, a JES job containing the FTP subcommands was submitted), the output will be sent to the JES joblog. Capturing the trace in this case requires simply retrieving the joblog itself. An additional destination for the FTP batch output can be specified using the 

 //SYSOUT DD

in the batch job's JCL.

FTP client traces can be started activated during client initialization, or after an FTP session has been started.

Activating tracing when FTP is initiated

Using the FTP.DATA file
Similar to the FTP server, tracing in the client can be activated by including any number of DEBUG statements in the FTP.DATA used by the client. Each option must be specified on a separate statement, as follows:

 DEBUG BAS   DEBUG SOC   DEBUG FSC

DUMP options can be set in the same manner:

 DUMP FSC   DUMP SOC   DUMP JES


Note: Specifying the statement

 TRACE

in the FTP.DATA file is the equivalent of specifying 

 DEBUG BAS

However, this is an outdated convention, and the use of the DEBUG statement is recommended.

Any FTP client using this FTP.DATA will be affected.

Using the (TRACE option
This option is the equivalent of specifying 

 DEBUG BAS

in the FTP.DATA file. If the FTP client is intiated in the TSO environment, this option can be specified using the syntax

 FTP remote_ftp_site (TRACE

However, if the FTP client is initiated in the UNIX System Services (USS) shell, the syntax becomes

 ftp remote_ftp_site / (TRACE


Using the '-d' option
Like the (TRACE option, the '-d' flag is the equivalent of setting 

 DEBUG BAS

in the FTP.DATA file. The TSO and USS syntax for using this flag is

 FTP -d remote_ftp_site


Concatenating trace options with a batch job
If the FTP client is being started in batch, and the JCL used to start the client has a SYSFTPD DD statement, DEBUG options can be concatenated into the FTP.DATA file. An example of doing this is as follows:

 //SYSFTPD DD DISP=SHR,DSN=SYS1.TCPPARMS(FTPDATA)   //        DD *   DEBUG ACC   DEBUG CMD   DEBUG FLO   /*


Activating tracing after the FTP client has already been started

Using the DEBUG subcommand
The DEBUG subcommand can be executed directly from the FTP command prompt to toggle tracing on and off. The output is sent to the user's screen, and is the equivalent of having otherwise activated trace points CMD, INT, FSC(1), and SOC(1). Executing the command a second time will set tracing to NONE.

Using the DEBUG subcommand with options
The DEBUG subcommand can also be executed using all of the trace options supported by the client, as described previously. All of the options can be specified on one command, such as

 DEBUG SOC(1) FSC(1) INT

or they can be specified on separate lines:

 DEBUG SOC(1)   DEBUG FSC(1)   DEBUG INT

In each case, the trace can be deactivated by executing the DEBUG subcommand with no options, or by specifying

 DEBUG NONE
Documentation needed for FTP TLS problems

  • FTP trace with DEBUG options SOC(3), SEC, CMD, FLO
  • System SSL API trace: The trace is turned on by setting the GSK_TRACE variable and optionally coding the GSK_TRACE_FILE. The default filename for GSK_TRACE_FILE is /tmp/gskssl.%.trc, where the % becomes the process ID. Set the GSK_TRACE variable to 0xFFFF. The following ENVAR statement should be added to the FTP server started proc or the FTP client's JCL:


    •   ENVAR("GSK_TRACE=0xFFFF")

    After obtaining the trace, format the trace file with the following OMVS command:

      gsktrace source_file > formatted_file

    Review the System SSL Programming manual for more information.

    Note: After collecting the System SSL trace and formatting it in the /tmp directory. It is recommended that this file be ftp'd to another system or moved to an MVS data set. The reason is that the /tmp data set is typically not a large data set, and if it becomes full it can cause applications that use the /tmp for their STDOUT to fail during initialization.
  • Use the RACF RACDCERT command to list information on Certificate being used. For example:


    •   RACDCERT ID(user) LIST

    will list certificate information. Similarly,

      RACDCERT ID(user) LISTRING(ringname)

    will list ring info.

    NOTE: 'ringname' is the name associated with the KEYRING keyword in the FTP.DATA file. 'user' is name of the certificate being used. For the default certificate for the FTP Server, the status must be set to TRUST.


Glossary of Acronyms
APARAuthorized Program Analysis Report
APIApplication Programming Interface
FTPFile Transfer Protocol
GDGGeneration Data Group
HFSHierarchical File System
IETFInternet Engineering Task Force
JCLJob Control Language
JESJob Entry Subsystem
LELanguage Environment
NATNetwork Address Translation
PMRProblem Management Record
PTFProgram Temporary Fix
RACFResource Access Control Facility
SQLStructured Query Language
SSLSecure Socket Layer
TLSTransport Layer Security

[{"Product":{"code":"SSSN3L","label":"z\/OS Communications Server"},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Component":"All","Platform":[{"code":"PF035","label":"z\/OS"}],"Version":"1.6;1.7;1.8;1.9;1.10;1.11;1.12;1.13;2.1;2.2","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}}]

Document Information

Modified date:
15 June 2018

UID

swg21318198

Page Feedback