Implementing client and server communications with the verbs interface (Linux only)

Developing applications that use the verbs interface requires a number of steps to create artifacts that are used in Remote Direct Memory Access (RDMA) connections between server and client. An example is provided to guide developers through the process on both the client and server sides of the connection.

Note: The RDMA implementation is deprecated and will likely be removed in a future release of IBM® SDK, Java™ Technology Edition, Version 8. A possible alternative is the open source Libfabric library.
The verbs API enables you to control how RDMA resources are created and used. Before you start any development work, you should have a good understanding of RDMA communications and the client and server environment for your Java application. The following types of RDMA resources are required:
  • Protection domains
  • Connection IDs
  • Queue pairs
  • Completion queues
You must determine how these resources should be associated with each other. For example, separate queue pairs can be associated either with a single completion queue, or with a shared completion queue. You must also determine how RDMA resources should be used for networking. For example, a completion queue can be polled directly, or you can request notifications on an event channel and poll the completion queue afterward.
The following diagram shows an example of a client and server implementation that uses the verbs API. Certain steps that are common to both the client and server processes are expanded in later diagrams. For a detailed description of the terms that are used in these communication flows, see jVerbs programming terms and artifacts (Linux only).
The diagram shows an example of the programming steps that are required for RDMA communication when you develop applications with the verbs API. Steps are required in a specific order on both the client and server sides of the connection. The communication events that are generated at specific stages of the process between the client and server system are also highlighted. For a complete description of the communication flow, read the associated text.
The following diagram expands the programming steps that are required when you allocate RDMA structures. This process is shown in the first diagram as the single step labeled "Allocate RDMA structures", which is marked with the number 1.
The diagram shows an example of the steps that are required to allocate RDMA structures for data transfer. These steps are required on both the client and server sides of the communication. For a description of the steps, read the associated text.
The following diagram expands the programming steps that are required when you remove RDMA structures. This process is shown in the first diagram as the single step labeled "Destroy allocated structures", which is marked with the number 2.
The diagram shows an example of the steps that are required to remove RDMA structures after data transfer. These steps are required on both the client and server sides of the communication. For a description of the steps, read the associated text.

The following sections provide an explanation of the server and client communication flows that are illustrated in the diagrams.

Server flow

The following events are established on the server side of the RDMA connection:
  1. An event channel is created.
  2. A connection ID is created and associated with the event channel. Any number of connection IDs can be associated with the event channel.
  3. The server listens for connection requests from clients.
  4. When a client connection request is received, the request is acknowledged. The event type for the request is RDMA_CM_EVENT_CONNECT_REQUEST.
  5. For each request received from a client the following steps occur:
    1. The server obtains the client connection ID.
    2. The necessary RDMA structures are allocated before the connection between the server and client is established. The following steps are required to create the RDMA structures:
      • The context for the device is obtained, which can be used to query the device, port, or global unique identifier (GUID).
      • A protection domain is allocated.
      • A completion channel is created for posting completion events.
      • A completion queue is created.
      • A work request is made for a completion queue notification.
      • A queue pair is created.
      • A direct byte buffer is allocated and registered for the data transfer.
    3. Optionally, a completion queue processing thread can be started. For more information about the events that take place, see Completion queue processing.
    4. When the RDMA structures are ready, a receive work request is posted by the server.
    5. When the work request is accepted, an event is sent to the client to confirm that the connection is established and ready to receive RDMA send or receive requests. The event type is RDMA_CM_EVENT_ESTABLISHED.
    6. A send or receive request is posted, which initiates data transfer between the server and client systems.
    7. When the work request is complete, the connection is disconnected. The event type RDMA_CM_EVENT_DISCONNECTED is generated by the server.
    8. The RDMA structures that were created for the data transfer are removed in the following order:
      1. The buffers are cleaned and unregistered.
      2. The completion queue is removed.
      3. The completion channel is removed.
      4. The queue pair is removed.
  6. To disconnect the server from further RDMA operations with client systems, the connection ID is removed.
  7. The event channel is removed. The event channel cannot be removed until all acknowledgments are received.

Client flow

The following events occur on the client side of the RDMA connection:
  1. An event channel is created.
  2. A connection ID is created and associated with the event channel. Any number of connection IDs can be associated with the event channel.
  3. The client queries the address of the server system by using the ConnectionID.ResolveAddress() method. When the event type RDMA_CM_EVENT_ADDRESS_RESOLVED is received, the client sends an acknowledgment.
  4. The client queries the route for the server system by using the ConnectionID.ResolveRoute() method. When the event type RDMA_CM_EVENT_ROUTE_RESOLVED is received, the client sends an acknowledgment.
  5. The necessary RDMA structures are allocated before the connection between the client and server is established. The following steps are required to create the RDMA structures:
    • The context for the device is obtained, which can be used to query the device, port, or global unique identifier (GUID).
    • A protection domain is allocated.
    • A completion channel is created for posting completion events.
    • A completion queue is created.
    • A work request is made for a completion queue notification.
    • A queue pair is created.
    • A direct byte buffer is allocated and registered for the data transfer.
  6. Optionally, a completion queue processing thread can be started. For more information about the events that take place, see Completion queue processing.
  7. A post receive request is made to the server.
  8. A connection request is made to the server. The event type RDMA_CM_CONNECT_REQUEST is generated and sent to the server.
  9. The client waits until the event type RDMA_CM_EVENT_ESTABLISHED is received from the server. This event indicates that the connection is established and data transfer is ready to take place.
  10. A send or receive work request is posted, which initiates data transfer between the server and client systems.
  11. When the work request is complete, the connection is disconnected. The event type RDMA_CM_EVENT_DISCONNECTED is generated by the client.
  12. The RDMA structures that were created for the data transfer are removed in the following order:
    1. The buffers are cleaned and unregistered.
    2. The completion queue is removed.
    3. The completion channel is removed.
    4. The queue pair is removed.
  13. To disconnect the client from further RDMA operations with the server, the connection ID is removed.
  14. The event channel is removed.

Completion queue processing

The following diagram expands the programming steps that are required when optionally processing completion queues. This process is shown in the first diagram as the single step labeled "Completion queue processing", which is marked with the number 3.
The diagram shows an example of the steps that are required to process completion queues. These steps are optional on both the client and server sides of the communication. For a description of the steps, read the associated text.
The following steps are shown in the diagram:
  1. The client or server uses the getCQEvent() and pollCQEvent() methods to retrieve the event of type RDMA_CM_EVENT ESTABLISHED from the event queue channel, which triggers processing.
  2. Work completion is processed.
  3. An acknowledgment is sent to the completion queue to confirm work completion.
  4. A request is made for a completion queue notification to ensure that the completion queue received the acknowledgement.