Programming tips for the verbs interface (Linux only)

Creating a stateful verbs method (SVM) object

Stateful operations avoid the performance impact of passing complex structures through the Java Native Interface (JNI) for fast-path data operations such as postSend() or postReceive(). You create the stateful method object only once but you can use the object multiple times. For example, you can send the same data again or receive data on the same buffers. The SVM object allocates direct byte buffers to hold the parameters that must be passed through the JNI. When an SVM object is no longer needed, you must free the SVM object by using the free() method so that the buffer can be marked for garbage collection. To find out whether the last operation was successful, use the IsSuccess() method.

Sharing completion queues

When completion queues are shared among multiple connection IDs, make sure that the completion queue entry is large enough for all send and receive operations. The limit is set by passing an argument when you create the queue by using the createCompletionQueue() method. If there are too many completion entries, an IOException exception occurs when the device limit is reached, and the remaining completions are missed.

The WorkCompletion.getQueuePairNum() method uniquely identifies the queue pair number that the work completion entry belongs to. Therefore, the work completion event can be dispatched to the corresponding connection ID to proceed with the operation.

Requesting notification

You must call the CompletionQueue.requestNotifyCQ() method to notify the application thread when the completion queue is waiting for a CompletionChannel.getCQEvent() call. If you do not request notification by using the CompletionQueue.requestNotifyCQ() method, a notification is not sent when a completion event is received on the completion queue. However, completion events are placed on the completion queue, which you can poll by using the pollCQ() method.

When a notification is received from a getCQEvent() method, you must call the requestNotifyCQ() method again for the next getCQEvent() method. You cannot call the CompletionQueue.requestNotifyCQ() method multiple times to queue requests; only one notification request is registered for the next getCQEvent() method.

The CompletionQueue.requestNotifyCQ() method accepts a Boolean data type that determines whether a notification is required on completion in the completion queue for solicited events, or for all events. If you want to mark an event as solicited, you can set the IBV_SEND_SOLICITED flag in the work request.

Creating queue pairs

A queue pair is required for posting send and receive requests. When you create a new queue pair, you must consider other active queue pairs that exist on the system. Choose a queue pair size that fits the number of work requests that are posted for send or receive at any point in time. Exceeding the device limit for a queue pair causes an IOException exception. Although you can set a queue pair size when the queue pair is created, the system can choose a more appropriate size that is based on existing queue pairs and available resource. To determine the actual size of a queue pair after creation, use the QueuePair.getQueuePairLimit() method.

Posting send or receive buffers

RDMA operations are asynchronous. Posting a send or receive buffer does not mean that the work is actually sent or received. The client or server that posted the send or receive buffer should wait for work completion to determine whether the send or receive operation was successful. You can query work completion status by using the WorkCompletion.getStatus() method to determine success or failure. You can also use the WorkCompletion.getOpcode() method to check for send or receive operation completion. If the work is complete, you can reuse the send or receive buffer. If the posted operation is not complete, reusing the same registered buffer can lead to a crash or memory corruption.

Buffers must be in place before you connect from the client side, and before you accept on the server side, to avoid missing any data transfer when the connection is established. The buffer position is not altered when data is received because the RDMA operation pins the buffer and copies the content. The application must handle changing the buffer position by determining the number of bytes that were written. Use the WorkCompletion.getByteLength() method to retrieve information about the number of bytes that were successfully sent or received. This data is applicable only when the completion status is successful.

Using the PollCQEvent() and getCQEvent() methods

The getCQEvent() method notifies the calling code when an event is received on the channel on which the method is called. You must call the PollCQEvent() method to retrieve the event from the channel. By pairing a getCQEvent() method with a pollCQEvent() method, you can check that an event is received then retrieve that event. Alternatively, you can use the pollCQEvent() method independently because a poll count of zero is returned if there is no event to retrieve. However, unlike the getCQEvent() method, the pollCQEvent() method occupies the processor, and uses processor cycles. Applications can choose to use the getCQEvent() and pollCQEvent() methods together, or just the pollCQEvent() method.

Removing resources