Status of Operations

To indicate the status of the engine and what actions the application should take, the SSLEngine.wrap() and SSLEngine.unwrap() methods return an SSLEngineResult instance, as shown in the previous examples. The SSLEngineResult contains two pieces of status information: the overall status of the engine and the handshaking status.

The possible overall statuses are represented by the SSLEngineResult.Status enum. Some examples of this status include OK, which means that there was no error, and BUFFER_UNDERFLOW, which means that the input buffer had insufficient data, indicating that the application needs to obtain more data from the peer (for example, by reading more data from the network), and BUFFER_OVERFLOW, which means that the output buffer had insufficient space to hold the result, indicating that the application needs to clear or enlarge the destination buffer.

Here is an example of how to handle BUFFER_UNDERFLOW and BUFFER_OVERFLOW statuses of SSLEngine.unwrap(). It uses SSLSession.getApplicationBufferSize() and SSLSession.getPacketBufferSize() to determine how large to make the byte buffers.

SSLEngineResult res = engine.unwrap(peerNetData, peerAppData);
switch (res.getStatus()) {
 case BUFFER_OVERFLOW:
  // Maybe need to enlarge the peer application data buffer.
  if (engine.getSession().getApplicationBufferSize() >
   peerAppData.capacity()) {
   // enlarge the peer application data buffer
  } else {
   // compact or clear the buffer
  }
  // retry the operation
  break;
 case BUFFER_UNDERFLOW:
  // Maybe need to enlarge the peer network packet buffer
  if (engine.getSession().getPacketBufferSize() >
   peerNetData.capacity()) {
   // enlarge the peer network packet buffer
  } else {
   // compact or clear the buffer
  }
  // obtain more inbound network data and then retry the operation
  break;
 // Handle other status: CLOSED, OK
 ...
}

The possible handshaking statuses are represented by the SSLEngineResult.HandshakeStatus enum. They represent whether handshaking has completed, whether the caller needs to obtain more handshaking data from the peer, send more handshaking data to the peer, and so on.

Having two statuses per result allows the engine to indicate that the application must take two actions: one in response to the handshaking and one representing the overall status of the wrap()/unwrap() method. For example, the engine might, as the result of a single SSLEngine.unwrap() call, return SSLEngineResult.Status.OK to indicate that the input data was processed successfully and SSLEngineResult.HandshakeStatus.NEED_UNWRAP to indicate that the application should obtain more SSL/TLS encoded data from the peer and supply it to SSLEngine.unwrap() again so that handshaking can continue. As you can see, the previous examples were greatly simplified; they would need to be expanded significantly to properly handle all of these statuses.

Here is an example of how to process handshaking data by checking handshaking status and the overall status of the wrap()/unwrap() method.

void doHandshake(SocketChannel socketChannel, SSLEngine engine,
ByteBuffer myNetData, ByteBuffer peerNetData) throws Exception {
 // Create byte buffers to use for holding application data
 int appBufferSize = engine.getSession().getApplicationBufferSize();
 ByteBuffer myAppData = ByteBuffer.allocate(appBufferSize);
 ByteBuffer peerAppData = ByteBuffer.allocate(appBufferSize);
 // Begin handshake
 engine.beginHandshake();
 SSLEngineResult.HandshakeStatus hs = engine.getHandshakeStatus();
 // Process handshaking message
 while (hs != SSLEngineResult.HandshakeStatus.FINISHED &&
  hs != SSLEngineResult.HandshakeStatus.NOT_HANDSHAKING) {
  switch (hs) {
   case NEED_UNWRAP:
    // Receive handshaking data from peer
    if (socketChannel.read(peerNetData) < 0) {
     // Handle closed channel
    }
    // Process incoming handshaking data
    peerNetData.flip();
    SSLEngineResult res = engine.unwrap(peerNetData, peerAppData);
    peerNetData.compact();
    hs = res.getHandshakeStatus();
    // Check status
    switch (res.getStatus()) {
     case OK :
      // Handle OK status
      break;
     // Handle other status: BUFFER_UNDERFLOW, BUFFER_OVERFLOW, CLOSED
     ...
    }
    break;
   case NEED_WRAP :
    // Empty the local network packet buffer.
    myNetData.clear();
    // Generate handshaking data
    res = engine.wrap(myAppData, myNetData);
    hs = res.getHandshakeStatus();
    // Check status
    switch (res.getStatus()) {
     case OK :
      myNetData.flip();
      // Send the handshaking data to peer
      while (myNetData.hasRemaining()) {
       if (socketChannel.write(myNetData) < 0) {
        // Handle closed channel
       }
      }
      break;
     // Handle other status: BUFFER_OVERFLOW, BUFFER_UNDERFLOW, CLOSED
     ...
    }
    break;
   case NEED_TASK :
    // Handle blocking tasks
    break;
   // Handle other status: // FINISHED or NOT_HANDSHAKING
   ...
  }
 }
 // Processes after handshaking
 ...
}