SSP: How to diagnose where a failure occurs

Body

Suggested initial steps in debugging a connection issue from an incoming client to a backend server through IBM® Sterling Secure Proxy.

IBM® Sterling Secure Proxy is a Demilitarized Zone (DMZ) based application proxy.  This usually places it (called the Secure Proxy Engine) behind the initial firewalls and ahead of the background "trusted" zones of an organization.  In addition, perimeter servers can be set up both in front of and behind the Sterling Secure Proxy server to add controlled entry and exit listening points in the other zones through encrypted tunnels.  This allows the the Security Administrators  to minimize the number of open connections through a firewall and also prevents direct access to the Sterling Secure Proxy server.  Configuration and monitoring of the engine is accomplished via a "Configuration Manager" (or CM) usually residing in the trusted zone.  Additional authentication may be provided using the Sterling External Authentication Server (SEAS or EA server), often permitting the authentication of users, certificates, and keys from external sources like LDAP and Active Directory.  Single Sign-on (SSO) capability may also be configured with servers in the trusted zone.

This general configuration provide a secure environment, but it can be difficult to diagnose.  Several common scenarios are:
a legitimate block due to expired certificates, invalid UserID/Passwords, etc.  
possible network or server outages
load balancers my be splitting or misdirecting traffic
all components not being brought up after planned restarts (Engine, Perimeter servers, External Authentication Servers and CM)

When diagnosing a connection error, several things must be kept in mind:
Sterling Secure Proxy provides session breaks in the DMZ to prevent direct access to internal systems
Remote trading partner sessions terminate in DMZ
A separate session from DMZ to trusted zone is established after successful authentication of trading partner
A separate connection is made to SEAS if configured

Communications paths:
Initially, new incoming client connections make contact at the external firewall at specific IP addresses.  Firewall  rules should be in place to allow a specified listening port at that address, and the routing of a connection specific servers.  The routed connection would usually be the front end perimeter server providing the listening points for the Sterling Secure Proxy Engine.  Secure Proxy will then authenticate the incoming client connection based on it's configuration.  If configured, this may open a connection to the SEAS Server, which might in turn open on to an LDAP, Active Directory or other authentication repository.  Once authenticated, the internal connection is made to the server(s) in the trusted zone (usually through another perimeter server).  We commonly call each communications path between two servers a 'leg'.  

This scenario provides us with several communications legs:
Client to firewall
firewall to front end Perimeter Server
front end Perimeter Server to Secure Proxy Engine
Secure Proxy Engine to SEAS Server
SEAS Server to LDAP or other authentication Server
Secure Proxy Engine to back-end perimeter server
back-end perimeter server to back-end servers

The first step you need to take is try to determine how far through the setup is the client connection making it?  You might want to start at the backend server and work backwards.  Are you seeing a connection attempt to the backend server?  Is the connection successful? Is any information transferred or is the session terminated immediately.  Is any information returned from the server?  If so, enable full logging on the backend server and see what error messages and warnings it is showing.  

Remember that if the connection fails due to authentication or any issue at the Secure Proxy area, then the backend connection won't even be attempted, so you won't see a connection coming in.  This would indicate that you need to start examining the SSP server next.

If you're seeing connection to the SSP engine but not to the backend server, then enable full logging on the SSP engine and SEAS to determine if the connection attempt is making it that far.  If you don't see a connection at all to Secure Proxy Engine, then you can be pretty sure that the connection is not going to the right port, is being blocked or mis-routed at the Firewall/Load balancer, or not communicating through the front end perimeter server.

Check to make sure the backend server address and port are correct and that they can be reached from the SSP engine or backend perimeter server as configured.  Examine the SSP logs. You should see messages that indicate if it did receive the client connection.  If authentication is enabled (either within SSP or SEAS) it should post a message indicating the type error.  Certificates, keys, UserID, passwords, and of course invalid information should be indicated in the logs.  If authentication is the failure point and you are using SEAS, then review the SEAS logs to determine what is failing.  This may point to invalid credentials, login failures, invalid certificates, or even connection failures to the LDAP, CRL, or other servers involved.