jVerbs programming terms and artifacts (Linux only)

The common terms are described in the following list. For more information, see the jVerbs API reference.

Completion channel

All completion events for send or receive posts are placed on a completion channel. You can use the CompletionChannel.getCQEvent() method to check when an event is placed on the completion channel. If the return is not null, a completion event is placed on a queue that is associated with the channel. You can retrieve the completion event by using the CompletionQueue.pollCQ() method. The CompletionChannel.getCQEvent() method does not indicate which completion queue to poll, therefore you must poll all the queues that are associated with the channel to retrieve the completion event.

Completion queue

Work completion events for send and receive posts are placed on a completion queue. Queues can be created for each connection ID, or queues can be shared among multiple connection IDs. When completion queues are shared, the work completion queue pair number uniquely identifies the connection ID that the work belongs to. When you create a completion queue, you can define the maximum number of completion events that can be held. A completion queue can be polled directly, or you can request notifications on an event channel and poll the completion queue afterward. If you poll the completion queue, you must poll at regular intervals to avoid missing any work completion events that might overrun.

Connection event

Connection management events are generated for different connection management calls. Here are some examples of common event types:

Connection request event: RDMA_CM_CONNECT_REQUEST

This event type is generated when a ConnectionID.connect() call is made from the active client side to the server side.

Established event: RDMA_CM_EVENT_ESTABLISED

This event type is generated when a connection is established on both sides.

Disconnection event: RDMA_CM_EVENT_DISCONNECTED

This event type is generated when a ConnectionID.destroy() call is made to the other side.

Address resolved event: RDMA_CM_EVENT_ADDRESS_RESOLVED

This event type is generated when a ConnectionID.resolveAdress() call is made from the active client side.

Route resolved event: RDMA_CM_EVENT_ROUTE_RESOLVED

This event type is generated when a ConnectionID.resolveRoute() call is made from the active client side.

For a complete list of event types, see the ConnectionEvent.EventType class in the API reference.

Connection ID

A connection ID is a unique identifier that identifies a connection between two RDMA systems. You create a connection ID by using the ConnectionID.create() method, passing the event channel as an argument to associate the connection ID with the channel. A connection event for a connection ID is received through an event channel.

Device context

The verbsContext class is specific to InfiniBand devices and describes the context in which resources can operate. Each InfiniBand device has one context; all the connection IDs that are associated with that device share the same context. Protection domains and completion channels are created by using device context.

Event channels

Event channels are used to receive all connection events for a connection ID. You can create an event channel by using the static method EventChannel.createEventChannel(). To get an event that was received on a channel, use the EventChannel.getConnectionEvent() method. To acknowledge an event, use the EventChannel.ackConnectionEvent() method.

Post send or receive request

A work request is posted to a queue pair that is associated with a connection ID. The work request might relate to a send or receive operation.

Protection domains

A protection domain controls the interaction between an RDMA adapter and host memory, by associating a queue pair and memory region. Components within a protection domain can interact only with each other.

Queue pair

Each connection ID must be associated with a queue pair to transfer data by using the ConnectionID.createQueuePair() method. All send or receive work requests that are generated by a connection ID are posted to that queue pair. Separate queue pairs can be associated either with a single completion queue, or with a shared completion queue.

ScatterGatherElement

A work request is made up of a list of ScatterGatherElement elements. Each element consists of the following information:

• A starting address of an allocated direct byte buffer that is to be sent, or the address location where remote data needs to be placed.
• The length of the buffer.
• A local key. The local key value corresponds to the local key value for the MemoryRegion instance that is returned by the registerMemoryRegion() method for that buffer.

Register and unregister memory

Any data buffers that are sent or received must first be registered to pin the memory for RDMA transfer. The buffers must also be unregistered after usage to unpin the memory. These tasks use the VerbsContext.registerMemoryRegion() and VerbsContext.deregisterMemoryRegion() API calls.

Work completion event

A work completion event is received on the completion queue for every send or receive work request. You can query the following details:

• Success or failure
• Number of bytes (read or write)
• Work request ID
• Queue pair number

When completion queues are shared between connection IDs, querying the queue pair number indicates which connection ID the work completion event is for.

Work request

There are two types of work requests:

• Send work request: This type of request is generated by a QueuePair.postSend() call. The request consists of an element, which is known as a ScatterGatherElement, a send flag, and an operation code. The ScatterGatherElement describes a local buffer.
• Receive work request: This type of request is generated by a QueuePair.postReceive() call. The request consists of a list of ScatterGatherElement elements.

You can post a list of work requests by using the ConnectionID.postSend() or ConnectionID.postReceive() methods.

A work completion event is received for each work request that is made.