gsk_secure_socket_init()

Initializes a secure socket connection.

Format

   #include <gskssl.h>

   gsk_status gsk_secure_socket_init(
                                      gsk_handle     soc_handle)

Parameters

soc_handle
Specifies the connection handle that is returned by the gsk_secure_socket_open() routine.

Results

The function return value is 0 (GSK_OK) if no error is detected. Otherwise, it is one of the return codes listed in the gskssl.h include file. These are some possible errors:
[GSK_CERTIFICATE_NOT_AVAILABLE]
No certificates available.
[GSK_ERR_3DES_KEYS_NOT_UNIQUE]
Triple DES key parts are not unique.
[GSK_ERR_BAD_CERT]
Certificate is not valid.
[GSK_ERR_BAD_CERTIFICATE_STATUS_RESPONSE]
Certificate status response is not valid.
[GSK_ERR_BAD_DATE]
Certificate is not valid yet or is expired.
[GSK_ERR_BAD_EC_PARAMS]
EC parameters are not supplied.
[GSK_ERR_BAD_HTTP_RESPONSE]
HTTP response is not valid.
[GSK_ERR_BAD_KEY_LABEL_LIST]
Key label list is not valid.
[GSK_ERR_BAD_KEYFILE_LABEL]
The specified key is not found in the key database or the key is not trusted.
[GSK_ERR_BAD_MAC]
Message verification failed.
[GSK_ERR_BAD_MESSAGE]
Incorrectly formatted message that is received from peer application.
[GSK_ERR_BAD_MSG_LEN]
Incorrectly formatted TLS extension data that is contained within message that is received from peer application.
[GSK_ERR_BAD_OCSP_RESPONSE]
OCSP response is not valid.
[GSK_ERR_BAD_PEER]
Peer application has violated the SSL protocol.
[GSK_ERR_BAD_PEER_ID]
Specified peer identifier is not valid.
[GSK_ERR_BAD_SID_VALUE]
Specified session identifier is not valid.
[GSK_ERR_BAD_SIG_ALG_PAIR]
Signature algorithm pairs list is not valid.
[GSK_ERR_BAD_V2_CIPHER]
SSL V2 cipher is not valid.
[GSK_ERR_BAD_V3_CIPHER]
SSL V3 cipher is not valid.
[GSK_ERR_BAD_V3_EXPANDED_CIPHER]
SSL V3 expanded cipher is not valid.
[GSK_ERR_CERT_VALIDATION]
Certificate validation error.
[GSK_ERR_CERTIFICATE_REVOKED]
Peer certificate is revoked.
[GSK_ERR_CLIENT_AND_SERVER_SID_DO_NOT_MATCH]
Client SID does not match the server SID.
[GSK_ERR_CRYPTO]
Cryptographic error is detected.
[GSK_ERR_EC_PARAMETERS_NOT_SUPPLIED]
EC parameters are not supplied.
[GSK_ERR_ECURVE_NOT_FIPS_APPROVED]
Elliptic Curve is not supported in FIPS mode.
[GSK_ERR_ECURVE_NOT_SUPPORTED]
Elliptic Curve is not supported.
[GSK_ERR_HOSTNAME_NOT_VALID]
HTTP server hostname is not valid.
[GSK_ERR_HTTP_IO_ERROR]
HTTP server communication error.
[GSK_ERR_HTTP_RESPONSE_TIMEOUT]
HTTP response is not received within configured time limit.
[GSK_ERR_ICSF_CLEAR_KEY_SUPPORT_NOT_AVAILABLE]
Clear key support not available due to ICSF key policy.
[GSK_ERR_ICSF_FIPS_DISABLED]
ICSF PKCS #11 services are disabled.
[GSK_ERR_ICSF_NOT_AVAILABLE]
ICSF services are not available.
[GSK_ERR_ICSF_NOT_FIPS]
ICSF PKCS #11 not operating in FIPS mode.
[GSK_ERR_ICSF_SERVICE_FAILURE]
ICSF callable service returned an error.
[GSK_ERR_INCOMPATIBLE_KEY]
Certificate key is not compatible with cipher suite.
[GSK_ERR_INCORRECT_KEY_ATTRIBUTE]
TKDS Private Key attributes do not support digital signature or RSA operation.
[GSK_ERR_INVALID_FRAGMENT_LENGTH]
An unsupported fragment length was received.
[GSK_ERR_IO]
I/O error communicating with peer application.
[GSK_ERR_KEY_IS_SMALLER_THAN_MINIMUM]
Peer's end-entity certificate key size is smaller than the minimum size allowed.
[GSK_ERR_LDAP]
An LDAP error is detected.
[GSK_ERR_LDAP_NOT_AVAILABLE]
The LDAP server is not available.
[GSK_ERR_LDAP_RESPONSE_TIMEOUT]
LDAP response is not received within configured time limit.
[GSK_ERR_MAX_RESPONSE_SIZE_EXCEEDED]
Maximum response size exceeded.
[GSK_ERR_MISSING_KEY_ALGORITHM]
Certificate key algorithm is not in signature algorithm pairs list.
[GSK_ERR_MISSING_SIGNATURE_ALGORITHM]
Signature algorithm is not in signature algorithm pairs list.
[GSK_ERR_MULTIPLE_DEFAULT]
Multiple keys are marked as the default.
[GSK_ERR_MULTIPLE_LABEL]
Multiple certificates exist for label.
[GSK_ERR_NO_CACHE_SESSION]
No cached session entry exists.
[GSK_ERR_NO_CERTIFICATE ]
No certificate received from partner.
[GSK_ERR_NO_CIPHERS]
No cipher specifications.
[GSK_ERR_NO_KEY_LABEL_LIST_MATCH]
There are no non-expired or valid key labels found that support the protocol, cipher, and session attributes.
[GSK_ERR_NO_PRIVATE_KEY]
Certificate does not contain a private key or the private key is unusable.
[GSK_ERR_NON_SUITE_B_CERTIFICATE]
Certificate does not meet Suite B requirements.
[GSK_ERR_OCSP_EXPIRED]
OCSP response is expired.
[GSK_ERR_OCSP_NONCE_CHECK_FAILED]
Nonce in OCSP response does not match value in OCSP request.
[GSK_ERR_OCSP_REQ_ERROR]
Error creating OCSP request.
[GSK_ERR_OCSP_RESPONDER_ERROR]
OCSP request failed with internal responder error.
[GSK_ERR_OCSP_RESPONSE_DUPLICATE_FOUND]
OCSP response contains duplicate certificate statuses.
[GSK_ERR_OCSP_RESPONSE_NOT_FOUND]
OCSP response does not contain requested certificate status.
[GSK_ERR_OCSP_RESPONSE_SIGALG_NOT_VALID]
OCSP response signature algorithm is not in the signature algorithm pairs list.
[GSK_ERR_OCSP_RESPONSE_TIMEOUT]
OCSP response was not received within configured time limit.
[GSK_ERR_OCSP_SIG_REQUIRED]
OCSP responder requires a signed request.
[GSK_ERR_PEER_ID_ATTRIBUTES_CONFLICT]
Client session cache attributes do not agree.
[GSK_ERR_REQ_CERT_BC_EXT_MISSING]
The required basic constraints certificate extension is missing.
[GSK_ERR_REVINFO_NOT_YET_VALID]
Revocation information is not yet valid.
[GSK_ERR_RNG]
Error encountered when generating random bytes.
[GSK_ERR_SECURE_LABEL_OPERATION_UNSUPPORTED]
A secure private key cannot be used with a fixed ECDH key exchange.
[GSK_ERR_SELF_SIGNED]
A self-signed certificate cannot be validated.
[GSK_ERR_SIGNATURE_NOT_SUPPLIED]
Signature not supplied.
[GSK_ERR_SOCKET_CLOSED]
Socket connection closed by peer application.
[GSK_ERR_UNKNOWN_CA]
A certification authority certificate is missing.
[GSK_ERR_UNRECOGNIZED_NAME]
The requested server name is not recognized.
[GSK_ERR_UNSUPPORTED]
SSL protocol or certificate type is not supported.
[GSK_ERR_UNSUPPORTED_CERTIFICATE_TYPE]
The certificate type is not supported by System SSL.
[GSK_ERR_UNSUPPORTED_REQUIRED_EXTENSION]
A required TLS extension has been rejected.
[GSK_ERR_UNSUPPORTED_EXTENSION]
An unrequested TLS Extension has been encountered.
[GSK_ERR_INAPPROPRIATE_PROTOCOL_FALLBACK]
An inappropriate protocol fallback is detected.
[GSK_INSUFFICIENT_STORAGE]
Insufficient storage is available.
[GSK_INVALID_HANDLE]
The connection handle is not valid.
[GSK_INVALID_STATE]
The connection is not in the open state or a previous initialization request has failed.
[GSK_RSA_TEMP_KEY_PAIR]
Unable to generate temporary RSA public/private key pair.
[GSK_WOULD_BLOCK_READ]
An attempt to read a handshake message failed with EWOULDBLOCK.
[GSK_WOULD_BLOCK_WRITE]
An attempt to write a handshake message failed with EWOULDBLOCK.

Usage

The gsk_secure_socket_init() routine initializes a secure socket connection created by the gsk_secure_socket_open() routine. After the connection has been initialized, it can be used for secure data transmission using the gsk_secure_socket_read() and gsk_secure_socket_write() routines. The gsk_secure_socket_close() routine should be called to close the connection when it is no longer needed. The gsk_secure_socket_close() routine should also be called if an error is returned by the gsk_secure_socket_init() routine.

Before calling the gsk_secure_socket_init() routine, the application must create a connected socket and store the socket descriptor in the SSL connection by calling the gsk_attribute_set_numeric_value() routine. For a client, this means calling the socket() and connect() routines. For a server, this means calling the socket(), listen(), and accept() routines. However, SSL does not require the use of TCP/IP for the communications layer. The socket descriptor can be any integer value which is meaningful to the application. The application must provide its own socket routines if it is not using TCP/IP by calling the gsk_attribute_set_callback() routine.

The gsk_secure_socket_init() routine can return GSK_WOULD_BLOCK_READ or GSK_WOULD_BLOCK_WRITE if the socket is in non-blocking mode. The connection is not initialized in this case and the application must call gsk_secure_socket_init() again when the socket is ready to accept a read request (GSK_WOULD_BLOCK_READ) or a write request (GSK_WOULD_BLOCK_WRITE). The application must provide its own callback routine for io_setsocketoptions() to have the SSL handshake processed in non-blocking mode (the default io_setsocketoptions() routine places the socket into blocking mode during the handshake processing).

In FIPS mode, only DSA certificates with domain parameters that conform to FIPS 186-3: Digital Signature Standard (DSS) are supported. In non-FIPS mode, if the key size is less than 1024 bits, then domain parameters that conform to FIPS 186-2 are supported. In non-FIPS mode, if the key size is greater than or equal to 1024 bits, the domain parameters must conform to FIPS 186-3, with the exception that parameters that have a prime modulus (p) of 2048 bits and a prime divisor (q) of 160 bits are also tolerated.

Be sure a gsk_secure_socket_shutdown() call is issued before a gsk_secure_socket_close() call.

Ensure that keys and algorithms are compliant for the chosen security strength when functioning at a FIPS mode level. For more information about FIPS mode level support, see System SSL and FIPS 140-2.

Protocol Selection
An SSL handshake is performed as part of the processing of the gsk_secure_socket_init() routine. This establishes the server identity and optionally the client identity. It also negotiates the cryptographic parameters to be used for the connection. The client and server attempts to use the highest available protocol version as determined by the intersection of the enabled protocol versions for the client and the server and the compatible ciphers. Thus:
  • TLS V1.2 is used if it is enabled on both the client and the server
  • If TLS V1.2 cannot be used and TLS V1.1 is enabled, negotiations drop back to TLS V1.1
  • If TLS V1.1 cannot be used and TLS V1.0 is enabled, negotiations drop back to TLS V1.0
  • If TLS V1.0 cannot be used and SSL V3 is enabled, negotiations drop back to SSL V3
  • If SSL V3 cannot be used, TLS V1.2 was not enabled on the client or server, and SSL V2 is enabled, negotiations drop back to SSL V2
Note:
  1. SSL V2 and SSL V3 are not as secure as TLS and should be disabled whenever possible to avoid attacks that force the client and server to drop back to SSL V2 or SSL V3 even though they are capable of using TLS V1.0, TLS V1.1, or TLS V1.2.
  2. When TLS extensions are defined for a client and any of the TLS protocols are enabled for the connection, SSL V2 is not negotiated even if it is enabled.
  3. If TLS V1.2 is enabled on the client, establishment of SSL sessions with SSL V2 servers is not supported.
Cipher selection

The client sends a list of ciphers it supports during the SSL handshake. The server application uses this list, and the defined ciphers that are supported by the server, to determine the cipher to be used during the SSL handshake. If the client is operating in FIPS mode, then the list provided only contains FIPS ciphers. A server executing in FIPS mode will only use FIPS ciphers. The cipher selection is done by looking through the servers cipher list for a match in the clients list. The first matching cipher is used.

When building the server's list of cipher suites for comparison with the list sent by the client, the server might omit some ciphers from the list as follows:
  • When executing in an export level cryptographic environment, any ciphers that are not permitted for use in an export level environment.
  • When executing in FIPS mode, any cipher suites that are not valid for use in FIPS mode.
  • Any cipher suites that specify a key algorithm that is not supported for use with the server certificate's key. For example, if the cipher requires an RSA key algorithm but the server certificate uses a DSA key algorithm.
  • When using protocol SSL V3.0 or earlier, any cipher suites that specify Elliptic Curve Cryptography.
  • When using protocol TLS V1.1 or earlier, any cipher suites that specify:
    • A sign key algorithm that is not supported for use with the server certificate's key. For example, if the cipher requires a Diffie-Hellman certificate signed with an RSA signature, but the server certificate is a Diffie-Hellman certificate that is signed with a DSA signature.
    • SHA-2 message authentication.
    • AES-GCM encryption.
  • When using protocol TLS V1.1 and higher, any cipher suites that specify 40-bit export encryption.
  • When using protocol TLS V1.2 and higher, any cipher suites that specify:
    • 56-bit DES encryption.
    • A key algorithm that is not specified in the signature algorithm pairs list that is supplied by the client.
Note: For protocols TLS V1.1 and higher, export cipher suites cannot be used. 40-bit ciphers is ignored if TLS V1.1 or higher is negotiated as the security protocol. If TLS V1.1 or higher is the intended protocol and only 40-bit ciphers are available, the connection fails with GSK_ERR_NO_CIPHERS.
Server certificate

The server certificate can use either RSA, DSA, Diffie-Hellman, or ECDSA as the public/private key algorithm.

In FIPS mode, the RSA or DSA key size must be at least 1024 bits, the Diffie-Hellman key size must be at least 2048 bits, and the ECC key size must be at least 192 bits and use a NIST-approved named curve.

An RSA certificate can be used with an RSA, ephemeral Diffie-Hellman, or ephemeral ECDH key exchange. A DSA certificate can be used with an ephemeral Diffie-Hellman key exchange. A Diffie-Hellman certificate can be used in a fixed Diffie-Hellman key exchange. An ECDSA certificate can be used with a fixed ECDH or ephemeral ECDH key exchange.

If the servers certificate contains a key usage extension during the SSL handshake, it must allow key usage as follows:
  • RSA certificates using export restricted ciphers (40-bit RC4 encryption and 40-bit RC2 encryption) with a public key size greater than 512 bits must allow digital signature. If operating in FIPS mode, export restricted ciphers cannot be selected.
  • Diffie-Hellman certificates that are used in fixed Diffie-Hellman key exchange must allow key agreement.
  • ECDSA certificates that are used in fixed ECDH key exchange must allow key agreement.
  • ECDSA certificates that are used in ephemeral ECDH key exchange must allow digital signature.
  • RSA certificates that are used in ephemeral ECDH key exchange must allow digital signature.
  • DSA certificates using ephemeral Diffie-Hellman key exchange must allow digital signature.
  • Other RSA certificates must allow key encipherment.

System SSL does not accept VeriSign Global Server ID certificates. When specified, System SSL uses these certificates as any other certificate when determining the encryption cipher to be used for the SSL session.

When using TLS V1.2 as the SSL session protocol, the client may pass to the server a list of signature algorithm pairs as part of the TLS handshake. The key algorithm and signature algorithm of the server certificate must be present in this list of signature algorithm pairs. In addition, any peer certificates in the server certificate chain must also be signed using a signature algorithm present in the list.

The signature algorithm pair list under the TLS V1.2 protocol may allow some TLS ciphers to operate using certificates that were previously incompatible with the cipher specification. In previous versions of TLS, these ciphers (primarily ciphers that use a fixed Diffie-Hellman or fixed ECDH key exchange) required the server certificate to be signed with a specific signature key algorithm. Under TLS V1.2, the signature algorithm pairs list allows the cipher to be used if the signature algorithm is specified in the list.

Client certificate

The SSL server always provides its certificate to the SSL client as part of the handshake. The client always performs server authentication using the certificate that is provided by the server. Server authentication requires that the provided certificate be validated. The validation process involves building the partner's certificate chain, using trusted certificates from the trusted certificate store and untrusted certificates that are provided through the handshake messages, as well as checking for certificate revocation information (LDAP CRLs, HTTP CRLs, or OCSP responses). See gsk_validate_certificate_mode() for a description how the certificate and revocation information is provided through data sources and the steps that are performed during certificate validation. Depending upon the server handshake type, the server may ask the client to provide its certificate. The key label that is stored in the connection is used to retrieve the certificate from the key database, PKCS #12 file, key ring, or token. The default key is used if no label is set. The key record must contain both an X.509 certificate and a private key. See gsk_validate_certificate_mode() for a description of the steps that are performed during certificate validation.

The client certificate can use either RSA, Digital Signature Standard algorithm (DSA), ECDSA, or Diffie-Hellman as the public/private key algorithm. The type of client certificate that can be used depends on the key exchange method being used for the session cipher that is selected by the server, as detailed in the following list:
  • RSA key exchange - RSA or DSA
  • Fixed Diffie-Hellman key exchange - Diffie-Hellman
  • Ephemeral Diffie-Hellman key exchange - RSA or DSA
  • Fixed ECDH key exchange - RSA, DSA, or ECDSA
  • Ephemeral ECDH key exchange - RSA, DSA, or ECDSA

Client certificates that are used in a fixed Diffie-Hellman or fixed ECDH key exchange where the client certificate is used to send the client's public key to the server must support key agreement. This means the certificate key usage extension (if any) must allow key agreement.

In all other cases the client certificate must support digital signatures. This means the certificate key usage extension (if any) must allow digital signature.

When using TLS V1.2 as the SSL session protocol, the server may pass to the client a list of signature algorithm pairs as part of the TLS handshake. The key algorithm and signature algorithm of the client certificate must be present in this list of signature algorithm pairs. In addition, any peer certificates in the client certificate chain must also be signed using a signature algorithm present in the list.

Key size and Diffie-Hellman group size support
When executing in non-FIPS mode the default minimum supported key sizes are:
  • Diffie-Hellman: 1024
  • DSA: 1024
  • ECC: 192
  • RSA: 1024
The default minimum key size can be overridden using attributes:
  • GSK_PEER_DH_MIN_KEY_SIZE
  • GSK_PEER_DSA_MIN_KEY_SIZE
  • GSK_PEER_ECC_MIN_KEY_SIZE
  • GSK_PEER_RSA_MIN_KEY_SIZE

Setting any of these attributes to a larger key size than is required for a FIPS level means that the peer must provide larger key sizes than required for the security strength. If the key size is set smaller than the required security strength, the weaker key may be accepted during early handshake processing, but the connection will fail later due to weak security strength of the key.

For ephemeral Diffie-Hellman key exchanges, the following attributes may be used to set minimum exchanged DHE group size:
GSK_CLIENT_EPHEMERAL_DH_GROUP_SIZE
  • LEGACY uses group size 1024 in non-FIPS mode and group size 2048 in FIPS mode.
  • 2048 always enforces group size 2048 regardless of FIPS mode.
GSK_SERVER_EPHEMERAL_DH_GROUP_SIZE
  • LEGACY uses group size 1024 in non-FIPS mode and group size 2048 in FIPS mode.
  • 2048 always enforces group size 2048 regardless of FIPS mode.
  • MATCH uses the DSA or RSA key size to define the required size to generate the Diffie-Hellman parameters. In non-FIPS mode if the key being matched is less than or equal to key size 1024, the group size 1024 is used. If the key being matched is greater than 1024, the group size 2048 is used. If executing in FIPS mode, the group size 2048 is used.
Specifying multiple labels

A server application can utilize environment variable GSK_SERVER_KEY_LABEL_LIST or specify attribute GSK_SERVER_KEYRING_LABEL_LIST for the environment or connection to specify a list of 1 to 8 key labels (certificate labels) to be chosen from in order to authenticate the server. The labels are specified in order of preference. The ability to specify more than one label allows for the server to negotiate with a wide range of clients that support varying ciphers.

During the negotiation of a secure connection, the first appropriate certificate is selected for use by the session based on the attributes for the session. The selection process uses the SSL attributes from both the client and server to make the decision. It is possible that no certificate is usable for a combination of attributes.

The selection order criteria is as follows:
  • Protocol
  • Cipher
  • Certificate

When the negotiated protocol is TLS V1.2, TLSV1.1, or TLS V1.0, or SSL V3, the server's cipher comparison list (see Cipher selection for information about how the server cipher comparison list is built) is processed. The cipher list is stepped through looking for a matching cipher suite supported by the client. When a matching cipher is encountered, the server's key label list is searched to locate the first non-expired certificate that supports the matching cipher suite and client supplied ecurves and signature algorithm session attributes when applicable. Additional validation will be performed against the selected certificate as the negotiation process continues. The specified server certificates should match the capabilities of the specified ciphers and session attributes. Should the server attempt to use a cipher that the certificate cannot accommodate, the connection will fail.

Specifying a cached session

When acting as a client, if the client intends to resume a newly created SSL V3, TLS V1.0, or higher session at a later time, immediately following the successful gsk_secure_socket_init() invocation to establish the connection, a call to gsk_attribute_get_buffer() needs to be performed in order to retrieve the peer ID (GSK_PEER_ID) for the newly generated connection. GSK_PEER_ID requires the enablement of the peer ID support by specifying environment attribute GSK_ENABLE_CLIENT_SET_PEERID to ON during the establishment of the SSL environment.

The GSK_PEER_ID value is used to locate a session entry in the client session cache so that it can be used to resume the session with the server if one exists and is not expired. If a matching session entry cannot be located by the server, a full handshake is performed.

The peer ID value provides information to locate the session information within the client session cache. Later, if the client wants to resume this session with the same server, as long as the SSL environment has not been closed, the client can set the peer ID to be used through a call to gsk_attribute_set_buffer() before calling gsk_secure_socket_init(). If the peer ID's cache entry is located, the following criteria is verified in order for this cache entry to be used:
  • The requested cipher must be included within the session's cipher list.
  • The requested protocol must be included within the session's supported protocol list.
  • The certificate labels must match.

If a cached session is required, set the GSK_REQ_CACHED_SESSION attribute to ON prior to calling gsk_secure_socket_init(). gsk_secure_socket_init() fails if the cached entry cannot be found. If the session ID value from a server hello does not match the cached session ID, gsk_secure_socket_init() fails.

If a cached session is not required, ensure that the GSK_REQ_CACHED_SESSION attribute is OFF (the default) prior to calling gsk_secure_socket_init(). This results in a full handshake if the cached entry cannot be found.

When acting as a server, if the next SSL V3, TLS V1.0, or higher connection from a particular client needs to utilize a previously established session, after the call to gsk_secure_socket_init() to establish the connection, call gsk_attribute_get_buffer() to retrieve the session ID (GSK_SID_VALUE) information for the newly generated connection. Before calling gsk_secure_socket_init() for the next connection, call gsk_attribute_set_buffer() to set the session ID value.

If the cached session is required, a server application does not require setting attribute GSK_REQ_CACHED_SESSION to ON to enforce the reusing of a cached session. By setting the GSK_SID_VALUE prior to calling gsk_secure_socket_init(), the server is required to locate that cached session.

The GSK_SID_VALUE value is used to locate a session entry in the server session cache that can be used to resume the session with a client if one exists and has not expired. gsk_secure_socket_init() fails if the cached entry cannot be found.

If the session ID value from a client hello does not match the cached session ID, gsk_secure_socket_init() fails.

Related topics