Contribute in GitHub: Edit online

HTTP Connection Issues

Problem

Two or more parties are involved in an HTTP connection.

• Client — initiates the connection

• Server — receives the connection

• Proxy — forwards the connection

Multiple proxies or hardware appliances can be chained together managing network traffic. These are invisible to the server, the client, or other proxies. They can be used for various reasons such as HTTPS termination, load balancing, or simply, to forward a connection between networks. Various proxy technologies are available such as:

• Self-hosted — Nginx, Apache, HAProxy, etc.

• Cloud — AWS Elastic Load Balancer, Google Cloud Load Balancing, Azure Load Balancer, etc.

• Hardware — F5, NetApp, PaloAlto, or a similar hardware appliance to manage network traffic

IBM Manta Data Lineage consists of interconnected services that communicate with each other over HTTP connections. The most common client and server variants are:

• CLI (client) → Flow Server (server)

• Admin GUI (client) → Flow Server (server)

• User browser (client) → Admin GUI (server)

• User browser (client) → Flow Server (server)

• CLI/AdminUI/FlowServer ↔︎ ConfigurationService

Sometimes a connection triggers an action on the server that takes longer to complete, such as the processing or export of a large amount of data. The connection can be prematurely interrupted by the client or proxies if they are not configured to allow connections that take longer to complete.

The impact on Manta Data Lineage varies, depending on which action triggered the connection exceptions. If the connection exceptions have broken a major functionality, the issue needs to be investigated ASAP. Even if the action that triggered the connection exceptions is unknown, it is still better to investigate the issue. Otherwise, the presence of the connection exceptions in the logs will complicate the investigation of any other issues by distracting from the focus of the investigation.

Symptoms

The CLI, Flow Server, or Admin GUI logs contain one of the following exceptions.

• org.apache.catalina.connector.ClientAbortException

• java.net.SocketTimeoutException

• java.io.IOException: Broken pipe

Or, the user receives one of the following exceptions in the browser.

• net::ERR_TIMED_OUT

• 504 Gateway Timeout

• Connection reset by peer

Solution

Steps:

• Check whether the connection exception can be matched in the client and server logs to identify which two parties are involved in the exception as the client and the server.

  • If there is a matching client and server log, it’s a Manta Data Lineage connection.

  • If there is a server log with no matching client log, it’s likely that the client is a user browser or a third-party application.

• If it’s a Manta Data Lineage connection:

  • Check the client configuration for server hostnames to identify if the services are connected to each other directly or via a proxy.

    • If the hostname is not localhost, it’s possible that there is a proxy.

    • If the hostname is localhost, it’s a direct connection.

  • Check the client timeout configuration and increase it as needed.

    • In the case of a load balancer, check the connection timeout and idle timeout periods.

• Check for potential proxies on the server hostname used for the connection.

• Check the proxy timeout configuration and increase it as needed.