Troubleshooting the proxy server
This topic helps you to solve problems that you might encounter with your proxy server.
About this task
Proxy server errors are logged in the joblog, Consult the following list if you are having problems with your proxy server.
Proxy server errors are logged in the SystemOut.log, proxy.log, or local.log files. Consult the following list if you are having problems with your proxy server.
Procedure
- The proxy server was created successfully, but I am
unable to start it. Check the SYSOUT file for port conflicts. Use the netstat -a command to see if any of the endpoints that are associated with the proxy server are already being used. You can find the ports in the administrative console by clicking Servers > Server Types > WebSphere proxy servers > server_name > Ports.If the proxy server fails to start when attempting to start it as a non-privleged user on UNIX systems, check for the following message in the logs:
ChannelFramew E CHFW0029E: Error initializing chain HTTPS_PROXY_CHAIN because of exception com.ibm.wsspi.channel.framework.exception.RetryableChannelException: Permission denied TCPPort E TCPC0003E: TCP Channel TCP_7 initialization failed. The socket bind failed for host * and port 80. The port may already be in use.
Change the ports of the PROXY_HTTP_ADDRESS and PROXY_HTTPS_ADDRESS transport chains to values greater than 1024. - The proxy server routes requests to the web container
over an administration port. The proxy server is located in front of several web containers. The configuration requires that the web containers listen to the non-default ports such as 9061, 9081, and so on. This scenario is the default case when multiple application servers are on the same machine, which forces new and different ports to be used in the configuration. In this scenario, the proxy server might route an application request to the web container over the administration port of 9061, instead of using the expected port of 9081.
Add the listening port numbers of the web container to the virtual host that is associated with the target application. This process will ensure that the proxy server routes the request to the web container over the correct port number.
- The proxy server started, but I am unable to access
the application resources through the endpoints for the proxy server. Ensure that the endpoints for the proxy server are among the host aliases in the virtual host that are associated with the application.
- The proxy server routes to another core group.
Verify that core group bridges exist between the core groups in the cell, and that the processes that are chosen to be bridges are restarted. If there is a firewall between the core group, verify that the correct ports are open for core group bridge traffic.
- The proxy server is unable to route requests to another
cell. Review the core group bridge settings. If the HMGR0149E error message is logged on any server, usually on a server acting as a core group bridge, then the security settings for the cell need to be modified to allow communication.
See the topic entitled Exporting Lightweight Third Party Authentication keys for more information about configuring cell security between multiple cells.
- Receiving a blank page when making a request to the
proxy. Consider the following actions:
- Update the virtual host. Ensure that the target application and routing rule are assigned to a virtual host that includes the proxy server listening ports (default: HTTP 80, HTTPS 443). Add the proxy server listening ports to the application, or routing rule virtual host, or use the proxy_host virtual host.
- Stop the conflicting process. Check your system to ensure that
no other process (for example, Apache, IBM HTTP Server, and so on) is running that uses the proxy server
ports (default: HTTP 80, HTTPS 443). If this problem occurs, the proxy
server seems to start normally, but is unable to receive requests
on the affected listening port. Check your system as follows:
- Stop the proxy server.
- Query your system using netstat and ps commands to determine if an offending process is using the port on which the proxy server is listening.
- If an offending process is found, stop the process and configure your system so that the process is not started during system startup.
- Enable proxy routing. Ensure that proxy routing is enabled for the web module of the application. Proxy routing is enabled by default, so if no proxy properties are modified, disregard this solution. Otherwise, see Customizing routing to applications for instructions on modifying the proxy properties.
- Test direct request. Ensure that the target application is installed by making a request directly to the application server. If a response is not received, then the problem is with the application server and not the proxy server. Verify this case by going through the proxy server after you can receive a response directly from the application server.
- HTTP 404 (File not found) error received from the proxy
server. Consider the following actions:
- Update the virtual host. Ensure that the target application and routing rule are assigned to a virtual host that includes the proxy server listening ports (default: HTTP 80, HTTPS 443). Add the proxy server listening ports to the application, or routing rule virtual host, or use the proxy_host virtual host.
- Enable proxy routing. Ensure that proxy routing is enabled for the web module of the application. Proxy routing is enabled by default, so if no proxy properties are modified, disregard this solution. Otherwise, see Customizing routing to applications for instructions on modifying the proxy properties.
- Test direct request. Ensure that the target application is installed by making a request directly to the application server. If a response is not received, then the problem is with the application server and not the proxy server. Verify this case by going through the proxy server when you can receive a response directly from the application server.
- Unable to make Secure Sockets Layer (SSL) requests to
application or routing rule. Ensure that the virtual host of the application or routing rule includes a host alias for the proxy server SSL port (default: 443).
- Unable to connect to the proxy server...request times
out. Stop the conflicting process. Check your system to ensure that no other process (for example, Apache, IBM HTTP Server, and so on) is running that uses the proxy server ports (default: HTTP 80, HTTPS 443). If this situation occurs, the proxy server seems to start normally, but is unable to receive requests on the affected listening port. Check your system, as follows:
- Stop the proxy server.
- Query your system using netstat and ps commands to determine if an offending process is using a port on which the proxy server is listening.
- If an offending process is found, stop the process and configure your system so that the process is not started during system startup.
-
Did not receive a response from the error page application when the HTTP error occurred (for
example, 404).
Ensure that the error page URI is entered correctly. Also, make sure that the Handle remote errors option is selected if you are handling HTTP error responses from back-end servers. For more detailed information, refer to Overview of the custom error page policy and the custom error page policy section of Proxy server settings.
- What packages do I enable when tracing the proxy server? All of the following packages are not needed for every trace, but if unsure, use all of them:
- *=info
- WebSphere Proxy=all
- GenericBNF=all
- HAManager=all
- HTTPChannel=all
- TCPChannel=all
- WLM*=all
- DCS=all
- ChannelFrameworkService=all
- com.ibm.ws.dwlm.*=all
- com.ibm.ws.odc.*=all
-
How do I enable SSL on/off load?
SSL on/off load is referred to as the transport protocol in the administrative console, and transport protocol is a web module property. Refer to Customizing routing to applications to see how to configure web module properties. No SSL on/off load or transport protocol properties exist for routing rules because the transport protocol is inherent to the generic server cluster that is specified in the routing rule.
-
When fronted by IBM HTTP Server or a web server
plug-in, how do I configure the proxy server so I do not have to add a port for it to the virtual
host?
For the proxy server to trust the security-related information that is included in a request, such as private headers that the product adds, add the originator of the request to the proxy server trusted security proxies list. For example, add an IBM HTTP Server or a web server plug-in that sends requests to the proxy server, to the proxy server trusted security proxies list. The web server plug-in can then send product added private header information that, contains the virtual host information of a request. If the proxy does not trust these private headers from the web server plug-in, or from any client, the proxy server adds its own private headers, which requires the addition of proxy server HTTP and HTTPS ports to the virtual host. Typically, when a web server plug-in is used with the proxy server, the intent is to use the proxy server as a back-end server. Therefore, you must add the plug-in as a trusted security proxy to avoid having to expose the proxy server ports. Routing requests from a plug-in to a proxy server provides more information about configuring a web server plug-in to use with the proxy server. Proxy server settings provides more information about setting up trusted security proxies.
- The proxy server seems to hang under stress, or Too
Many Files Open exceptions display in ffdc or SystemErr.log.
Under high connection loads, the number of file system descriptors might become exhausted and the proxy server may seem to hang and drop Too Many Files Open exceptions in the ffdc directory or in the SystemError.log file because it is unable to open a socket. To alleviate this problem, modify one or more of the following parameters at the operating system level, and at the proxy server level to optimize the use of connections for the proxy server:
- Operating system tuning for Windows 2003, and XP
- TcpTimedWaitDelay - Determines the time that must elapse before TCP/IP releases a closed
connection and reuse its resources. This interval between closure and release is known as the
TIME_WAIT state, or twice the maximum segment lifetime (2MSL) state. During this time, reopening the
connection to the client and server costs less than establishing a new connection. By reducing the
value of this entry, TCP/IP releases closed connections faster and can provide more resources for
new connections. Adjust this parameter if the running application requires rapid release, the
creation of new connections, or an adjustment because of a low throughput caused by multiple
connections in the TIME_WAIT state. View or set this value as follows:
- Use the regedit command and access the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\TCPIP\Parameters registry subkey to create a new REG_DWORD value named TcpTimedWaitDelay.
- Set the value to decimal 30, which is Hex 0x0000001e. This value sets the wait time to 30 seconds.
- Stop and restart your system.
Information Value Default value 0xF0, which sets the wait time to 240 seconds (4 minutes). Recommended value A minimum value of 0x1E, which sets the wait time to 30 seconds. - MaxUserPort - Determines the highest port number that TCP/IP can assign when an application
requests an available user port from the system. View or set this value as follows:
- Use the regedit command, access the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\TCPIP\Parameters registry subkey, and create a new REG_DWORD value named MaxUserPort.
- Set this value to at least decimal 32768.
- Stop and restart your system.
Information Value Default value None Recommended value At least decimal 32768.
- TcpTimedWaitDelay - Determines the time that must elapse before TCP/IP releases a closed
connection and reuse its resources. This interval between closure and release is known as the
TIME_WAIT state, or twice the maximum segment lifetime (2MSL) state. During this time, reopening the
connection to the client and server costs less than establishing a new connection. By reducing the
value of this entry, TCP/IP releases closed connections faster and can provide more resources for
new connections. Adjust this parameter if the running application requires rapid release, the
creation of new connections, or an adjustment because of a low throughput caused by multiple
connections in the TIME_WAIT state.
- Operating system tuning for Linux®
- timeout_timewait parameter - Determines the time that must elapse before TCP/IP releases a
closed connection and can reuse its resources. This interval between closure and release is known as
the TIME_WAIT state, or twice the maximum segment lifetime (2MSL) state. During this time, reopening
the connection to the client and server costs less than establishing a new connection. By reducing
the value of this entry, TCP/IP can release closed connections faster, providing more resources for
new connections. Adjust this parameter if the running application requires rapid release, the
creation of new connections, and a low throughput due to many connections sitting in the TIME_WAIT
state. View or set this value by issuing the following command to set the timeout_timewait parameter
to 30 seconds:
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
- Linux file descriptors (ulimit) - Specifies the number
of open files that are supported. The default setting is typically sufficient for most applications.
If the value set for this parameter is too low, a file open error, memory allocation failure, or
connection establishment error might display.
View or set this value by checking the UNIX reference pages on the ulimit command for the syntax of different shells. Set the ulimit command to 65535 for the KornShell shell (ksh), by issuing the ulimit -n 65535 command. Use the ulimit -a command to display the current values for all limitations on system resources.
Information Value Default value 1024 Recommended value 65535
- timeout_timewait parameter - Determines the time that must elapse before TCP/IP releases a
closed connection and can reuse its resources. This interval between closure and release is known as
the TIME_WAIT state, or twice the maximum segment lifetime (2MSL) state. During this time, reopening
the connection to the client and server costs less than establishing a new connection. By reducing
the value of this entry, TCP/IP can release closed connections faster, providing more resources for
new connections. Adjust this parameter if the running application requires rapid release, the
creation of new connections, and a low throughput due to many connections sitting in the TIME_WAIT
state. View or set this value by issuing the following command to set the timeout_timewait parameter
to 30 seconds:
- Operating system tuning for AIX®
- TCP_TIMEWAIT - Determines the time that must elapse before TCP/IP releases a closed connection
and can reuse its resources. This interval between closure and release is known as the TIME_WAIT
state, or twice the maximum segment lifetime (2MSL) state. During this time, reopening the
connection to the client and server costs less than establishing a new connection. By reducing the
value of this entry, TCP/IP can release closed connections faster, providing more resources for new
connections. Adjust this parameter, if the running application requires rapid release or the
creation of new connections, or if a low throughput occurs due to many connections sitting in the
TIME_WAIT state. View or set this value by issuing the following command to set the TCP_TIMEWAIT
state to 15 seconds:
/usr/sbin/no -o tcp_timewait =1
- AIX file descriptors (ulimit) - Specifies the number of
open files that are permitted. The default setting is typically sufficient for most applications. If
the value set for this parameter is too low, errors can occur when opening files or establishing
connections, and a memory allocation error might display. To prevent WebSphere® Application Server from running short on resources, remove the
upper limits (ulimit) for resources on the user account on which the WebSphere Application Server process runs.View or set this value by changing the ulimit settings as follows:
- Open the command window.
- Type smitty users to open the AIX configuration program.
- Select Change or Show Characteristics of a user.
- Type the name of the user account on which the WebSphere Application Server runs.
- Press Enter.
- Change the following settings to the indicated value:
Information Value Soft FILE Size -1 Soft CPU Time -1 Soft STACK Size -1 Soft CORE File Size -1 Hard FILE Size -1 Hard CPU Time -1 Hard STACK Size -1 Hard CORE File Size -1 - Press Enter to save changes.
- Log out and log into your account.
- Restart the product.
Information Value Default value 2000 Recommended value unlimited
- TCP_TIMEWAIT - Determines the time that must elapse before TCP/IP releases a closed connection
and can reuse its resources. This interval between closure and release is known as the TIME_WAIT
state, or twice the maximum segment lifetime (2MSL) state. During this time, reopening the
connection to the client and server costs less than establishing a new connection. By reducing the
value of this entry, TCP/IP can release closed connections faster, providing more resources for new
connections. Adjust this parameter, if the running application requires rapid release or the
creation of new connections, or if a low throughput occurs due to many connections sitting in the
TIME_WAIT state. View or set this value by issuing the following command to set the TCP_TIMEWAIT
state to 15 seconds:
- Operating system tuning for HP-UX
HP-UX nfile kernel parameter - Specifies the maximum number of files that can be open on the system at any given time. Not specifying a large enough number might restrict your system processing capacity.
HP-UX ninode kernel parameter - Specifies the maximum number of open inodes that can be in memory. An open inode is associated with each unique open file. Therefore, the value you specify for the ninode parameter should be larger than the number of unique files that must be open at the same time.
- Operating system tuning for Solaris
- Solaris file descriptors (ulimit) - Specifies the number of open files that are supported. If
the value specified for this parameter is too low, a file open error, memory allocation failure, or
connection establishment error might display. View or set this value by checking the UNIX reference pages on the ulimit command for the syntax of
different shells.
- Issue the
ulimit -a
command to display the current values for all limitations on system resources. - Issue the
ulimit -n 65535
command to set the ulimit command to 65535 for the KornShell shell (ksh). - The maximum number of files that can be open for a process is also affected by the
rlim_fd_max
, andrlim_fd_cur
settings for the global kernel limit. You might need to increase the values of these settings in the /etc/system file.
Solaris 10 provides a new mechanism for setting kernel parameters. Issue one of the following commands to specify a new limit for the max number of file descriptor kernel parameter on Solaris 10:- To change the value for the current shell session:
prctl -n process.max-file-descriptor -r -v 65535 $$
- To make the change a system wide
change:
projmod -sK 'process.max-file-descriptor=(privileged,65535,deny)' system
Caution should be used when issuing this command because this command changes the setting for all users, and all projects.
- To change the value for all projects that are owned by user root:
projmod -sK 'process.max-file-descriptor=(privileged,65535,deny)' user.root
- To change the value for all projects owned by non-root user
projmod -sK 'process.max-file-descriptor=(privileged,65535,deny)' user.username
- Issue the
- TCP_TIME_WAIT_INTERVAL - Notifies TCP/IP on how long to keep the connection control blocks
closed. After the applications complete the TCP/IP connection, the control blocks are kept for the
specified time. When high connection rates occur, a large backlog of the TCP/IP connections
accumulate and can slow server performance. The server can stall during certain peak periods. If the
server stalls, the netstat command shows that many of the sockets that are opened to the HTTP
server are in the CLOSE_WAIT or FIN_WAIT_2 state. Visible delays can occur for up to four minutes,
during which time the server does not send any responses, but CPU utilization stays high with all of
the activities in system processes. View or set this value by using the get command to determine the current interval and the set command to specify an interval of 30 seconds. For example:
ndd -get /dev/tcp tcp_time_wait_interval ndd -set /dev/tcp tcp_time_wait_interval 30000
Information Value Default value 240000 milliseconds, which is equal to 4 minutes. Recommended value 60000 milliseconds
- Solaris file descriptors (ulimit) - Specifies the number of open files that are supported. If
the value specified for this parameter is too low, a file open error, memory allocation failure, or
connection establishment error might display. View or set this value by checking the UNIX reference pages on the ulimit command for the syntax of
different shells.
- Proxy server tuning
- Persistent requests - A persistent request is one that is sent over an existing TCP connection.
You can maximize performance by increasing the number of requests that are received over a TCP
connection from a client. The value should represent the maximum number of embedded objects, for
instance GIF and so on, in a web page +1.
View or set this value in the WebSphere Application Server administrative console by clicking Servers > Proxy Servers > server_name > Proxy server transports > HTTP_PROXY_CHAIN/HTTPS_PROXY_CHAIN.
Information Value Default value 100 Recommended value A value that represents the maximum number of embedded objects in a web page + 1. - Outbound connection pool size - The proxy server pools outbound connections to target servers
and the number of connections that resides in the pool is configurable. If the connection pool is
depleted or empty, the proxy server creates a new connection to the target server. Under high
concurrent loads, increase the connection pool size should to a value of the expected concurrent
client load to achieve optimal performance.
View or set this value in the WebSphere Application Server administrative console by clicking Servers > Proxy Servers > server_name > Proxy server transports > HTTP Proxy Server Settings. In the Content Server Connection section, increase the maximum connections per server field to a value that is equal to or greater than the expected maximum number of connected clients. Save your changes, synchronize the changes to the proxy server node, and restart the proxy server.
Information Value Recommended value Value consistent to the expected concurrent client load. - Outbound request time-out - Often times, the back-end application servers that are fronted by
the proxy server may be under high load and may not respond in an adequate amount of time, therefore
the connections on the proxy server may be tied up from waiting for the back-end application server
to respond. Alleviate this by configuring the amount of time the proxy server waits for a response
from the target server. This is the Outbound Request Time-out value. By managing the amount of time
the proxy server waits for a slow back-end application server, connections are freed up faster and
used for other request work.View or set this value in the administrative console by clicking Servers > Server Types > WebSphere proxy servers > proxy_server_name > HTTP Proxy Server Settings. In the Content Server Connection section, set Outbound Request Time-out to a value that represents the acceptable response time from the point of view of the client.
Information Value Default value 120 Recommended value A value that represents the acceptable response time from the point of view of the client.
- Persistent requests - A persistent request is one that is sent over an existing TCP connection.
You can maximize performance by increasing the number of requests that are received over a TCP
connection from a client. The value should represent the maximum number of embedded objects, for
instance GIF and so on, in a web page +1.
- Operating system tuning for Windows 2003, and XP