Using delegation token endpoints

The delegation token endpoints are used to fetch a delegation token for HDFS, Hive, and Hive Metastore (HMS) services. The delegation tokens can then be used for authentication when you access the specific Hadoop services, such as HDFS, Hive and HMS.

Along with the delegation token, the endpoint also returns certain configuration of the services that can be used in any way by the client.

Important: Your Hadoop systems must be Kerberos-enabled for the endpoints to work. An error message appears if your cluster isn't Kerberos-enabled: `The request failed due to unsupported authentication type in the cluster.`

If a Hive service is exposed on a Hadoop cluster, the Hive configuration hive.server2.enable.doAs should be enabled or set to true for the service. If the configuration isn't enabled, this Hive impersonation is turned off and the Hive operations will be executed as the "hive" user instead of the user submitting the query.

Endpoint examples

The following examples are for the HDFS service. If you want to see examples for Hive or HMS, replace HDFS with Hive or HMS in the following examples.

Endpoint to get delegation tokens of Hive, HDFS, and HMS all together

Note: You can also fetch the delegation token of only one service at a time by providing only that service's name in the services query parameter in the following URL.

For example:

curl -k -H "authorization: Bearer $TOKEN" -X POST "https://shadedgepn1.fyre.ibm.com:8443/gateway/wslpatch12-master-1/dsxhi/v1/delegationToken?doAs=user1&services=hdfs,hms,hive"

{"tokens":
["delegationToken":"NAAFdXNlcjEFZHN4aGkSZHN4aGlARllSRS5JQk0uQ09NigFz_1WBFIoBdCNiBRSOIcKOA2YUIL4Lk3BvwhlTZOMNP4G5TrTgmLMVSERGU19ERUxFR0FUSU9OX1RPS0VOD2hhLWhkZnM6bWFnaWNucw","service":"hdfs","config":{"fs.defaultFS":"hdfs://magicns"}},
{"delegationToken":"MgAFdXNlcjEFZHN4aGkSZHN4aGlARllSRS5JQk0uQ09NigFz_1WCVYoBdCNiBlUCjgMxFAWWfq5O58FNl7t2PMu0Z4HoFk8WFUhJVkVfREVMRUdBVElPTl9UT0tFTgA","service":"hms","config":

{"hive.metastore.sasl.enabled":"true","hive.metastore.uris":"thrift://shad2.fyre.ibm.com:9083","hive.metastore.kerberos.principal":"hive/_HOST@FYRE.IBM.COM"}},
{"delegationToken":"JQAFdXNlcjEFZHN4aGkFZHN4aGmKAXP_VYUYigF0I2IJGAGOA28UDB3AP8JJBYb_5BYe6vlulx14IRcVSElWRV9ERUxFR0FUSU9OX1RPS0VOFmhpdmVzZXJ2ZXIyQ2xpZW50VG9rZW4","service":"hive","config":

{"method":"kerberos","jdbcUrl":"jdbc:hive2://shad2.fyre.ibm.com:10000/;AuthMech=6;DelegationToken=DELEGATION_TOKEN"}}]}

Endpoint to renew HDFS delegation token

Every delegation token has an expiration date and time. The same delegation token can be renewed before its expiration so that it can be reused. To renew the delegation token, it must be passed as part of the renew endpoint's request header.

Provide the service name as a query parameter, such as service=hdfs in the following example, to which the delegation token you are renewing belongs to. You can renew a delegation token for only one service at a time.

For example:

export
delegationToken=NAAFdXNlcjEFZHN4aGkSZHN4aGlARllSRS5JQk0uQ09NigFz_1WBFIoBdCNiBRSOIcKOA2YUIL4Lk3BvwhlTZOMNP4G5TrTgmLMVSERGU19ERUxFR0FUSU9OX1RPS0VOD2hhLWhkZnM6bWFnaWNucw

curl -k -H "authorization: Bearer $TOKEN" -H "X-DelegationToken: $delegationToken" -X PUT "https://shadedgepn1.fyre.ibm.com:8443/gateway/wslpatch12-master-1/dsxhi/v1/delegationToken?doAs=user1&service=hdfs"

Endpoint to delete or cancel the delegation token

If the delegation token is not needed or not good to be used anymore, then it can be deleted. To delete the delegation token, it must be passed as part of the delete endpoint's request header.

Provide the service name as a query parameter, such as service=hdfs in the following example, to which the delegation token you are deleting belongs to. You can cancel a delegation token for only one service at a time.

For example:

export delegationToken=NAAFdXNlcjEFZHN4aGkSZHN4aGlARllSRS5JQk0uQ09NigFz_1WBFIoBdCNiBRSOIcKOA2YUIL4Lk3BvwhlTZOMNP4G5TrTgmLMVSERGU19ERUxFR0FUSU9OX1RPS0VOD2hhLWhkZnM6bWFnaWNucw`

curl -k -H "authorization: Bearer $TOKEN" -H "X-DelegationToken: $delegationToken" -X DELETE "https://shadedgepn1.fyre.ibm.com:8443/gateway/wslpatch12-master-1/dsxhi/v1/delegationToken?doAs=user1&service=hdfs"`

Resolving errors

When your renewal or deletion of delegation token endpoints are successful, a 204 status code and empty content are returned.

When there is an error, the endpoints return a response with _statusCode_ field as 500 and the message field contains the full error stack-trace.

Note: Different services produce different error messages. The Message field contains the response. The `statusCode` field in the response for this generic type of errors will always be 500. Besides the generic errors, the other type of errors will have different values in the response's `statusCode` field.

For example, the following error message is displayed with a status code after you tried to renew a token that was deleted or canceled:

curl -k -H "authorization: Bearer $TOKEN" -H "X-DelegationToken: $delegationToken" -X PUT "https://shadedgepn1.fyre.ibm.com:8443/gateway/wslpatch12-master-1/dsxhi/v1/delegationToken?doAs=user1&service=hdfs"

{"exception":null,"_statusCode_":500,"message":"
<MessageNotFound::org.apache.hadoop.security.token.SecretManager$InvalidToken: Renewal
request for unknown token\n\tat
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToke
n(AbstractDelegationTokenSecretManager.java:502)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewDelegationToken(FSNamesystem
.java:7171)\n\tat
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewDelegationToken(NameNo
deRpcServer.java:683)\n\tat
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewDeleg
ationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:1028)\n\tat
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtoco
l$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.j
ava:640)\n\tat
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)\n\tat
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)\n\tat
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)\n\tat
java.security.AccessController.doPrivileged(Native Method)\n\tat
javax.security.auth.Subject.doAs(Subject.java:422)\n\tat
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)\n\tat
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)\n>"}

Limitations

When you renew an HMS delegation token for HDP 2.x and CDH 5.x Hadoop clusters, for example:

curl -k -H "authorization: Bearer $TOKEN" -H "X-DelegationToken: $delegationToken" -X PUT "https://shadedgepn1.fyre.ibm.com:8443/gateway/wslpatch12-master-1/dsxhi/v1/delegationToken?doAs=user1&service=hms"

The following error message appears:

{"exception":null,"_statusCode_":500,"message":"<MessageNotFound::MetaException(message:dsxhi@FYRE.IBM.COM tries to renew a token with renewer dsxhi)>"}

This is a limitation of the Hive version in HDP 2.x and CDH 5.x. Hadoop clusters. This limitation was fixed in Hive versions 3.0.0 or later. For more information about Hive Jira, see: https://issues.apache.org/jira/browse/HIVE-16708:

Parent topic: Apache Hadoop