Using delegation token endpoints
The delegation token endpoints are used to fetch a delegation token for HDFS, Hive, and Hive Metastore (HMS) services. The delegation tokens can then be used for authentication when you access the specific Hadoop services, such as HDFS, Hive and HMS.
Along with the delegation token, the endpoint also returns certain configuration of the services that can be used in any way by the client.
If a Hive service is exposed on a Hadoop cluster, the Hive configuration hive.server2.enable.doAs
should be enabled or set to true for the service. If the configuration isn't enabled, this Hive impersonation is turned off and the
Hive operations will be executed as the "hive" user instead of the user submitting the query.
Endpoint examples
The following examples are for the HDFS service. If you want to see examples for Hive or HMS, replace HDFS with Hive or HMS in the following examples.
Endpoint to get delegation tokens of Hive, HDFS, and HMS all together
For example:
curl -k -H "authorization: Bearer $TOKEN" -X POST "https://shadedgepn1.fyre.ibm.com:8443/gateway/wslpatch12-master-1/dsxhi/v1/delegationToken?doAs=user1&services=hdfs,hms,hive"
{"tokens":
["delegationToken":"NAAFdXNlcjEFZHN4aGkSZHN4aGlARllSRS5JQk0uQ09NigFz_1WBFIoBdCNiBRSOIcKOA2YUIL4Lk3BvwhlTZOMNP4G5TrTgmLMVSERGU19ERUxFR0FUSU9OX1RPS0VOD2hhLWhkZnM6bWFnaWNucw","service":"hdfs","config":{"fs.defaultFS":"hdfs://magicns"}},
{"delegationToken":"MgAFdXNlcjEFZHN4aGkSZHN4aGlARllSRS5JQk0uQ09NigFz_1WCVYoBdCNiBlUCjgMxFAWWfq5O58FNl7t2PMu0Z4HoFk8WFUhJVkVfREVMRUdBVElPTl9UT0tFTgA","service":"hms","config":
{"hive.metastore.sasl.enabled":"true","hive.metastore.uris":"thrift://shad2.fyre.ibm.com:9083","hive.metastore.kerberos.principal":"hive/_HOST@FYRE.IBM.COM"}},
{"delegationToken":"JQAFdXNlcjEFZHN4aGkFZHN4aGmKAXP_VYUYigF0I2IJGAGOA28UDB3AP8JJBYb_5BYe6vlulx14IRcVSElWRV9ERUxFR0FUSU9OX1RPS0VOFmhpdmVzZXJ2ZXIyQ2xpZW50VG9rZW4","service":"hive","config":
{"method":"kerberos","jdbcUrl":"jdbc:hive2://shad2.fyre.ibm.com:10000/;AuthMech=6;DelegationToken=DELEGATION_TOKEN"}}]}
Endpoint to renew HDFS delegation token
Every delegation token has an expiration date and time. The same delegation token can be renewed before its expiration so that it can be reused. To renew the delegation token, it must be passed as part of the renew endpoint's request header.
Provide the service name as a query parameter, such as service=hdfs
in the following example, to which the delegation token you are renewing belongs to. You can renew a delegation token for only one service at a time.
For example:
export
delegationToken=NAAFdXNlcjEFZHN4aGkSZHN4aGlARllSRS5JQk0uQ09NigFz_1WBFIoBdCNiBRSOIcKOA2YUIL4Lk3BvwhlTZOMNP4G5TrTgmLMVSERGU19ERUxFR0FUSU9OX1RPS0VOD2hhLWhkZnM6bWFnaWNucw
curl -k -H "authorization: Bearer $TOKEN" -H "X-DelegationToken: $delegationToken" -X PUT "https://shadedgepn1.fyre.ibm.com:8443/gateway/wslpatch12-master-1/dsxhi/v1/delegationToken?doAs=user1&service=hdfs"
Endpoint to delete or cancel the delegation token
If the delegation token is not needed or not good to be used anymore, then it can be deleted. To delete the delegation token, it must be passed as part of the delete endpoint's request header.
Provide the service name as a query parameter, such as service=hdfs
in the following example, to which the delegation token you are deleting belongs to. You can cancel a delegation token for only one service at a time.
For example:
export delegationToken=NAAFdXNlcjEFZHN4aGkSZHN4aGlARllSRS5JQk0uQ09NigFz_1WBFIoBdCNiBRSOIcKOA2YUIL4Lk3BvwhlTZOMNP4G5TrTgmLMVSERGU19ERUxFR0FUSU9OX1RPS0VOD2hhLWhkZnM6bWFnaWNucw`
curl -k -H "authorization: Bearer $TOKEN" -H "X-DelegationToken: $delegationToken" -X DELETE "https://shadedgepn1.fyre.ibm.com:8443/gateway/wslpatch12-master-1/dsxhi/v1/delegationToken?doAs=user1&service=hdfs"`
Resolving errors
When your renewal or deletion of delegation token endpoints are successful, a 204 status code and empty content are returned.
When there is an error, the endpoints return a response with _statusCode_ field
as 500
and the message field contains the full error stack-trace.
For example, the following error message is displayed with a status code after you tried to renew a token that was deleted or canceled:
curl -k -H "authorization: Bearer $TOKEN" -H "X-DelegationToken: $delegationToken" -X PUT "https://shadedgepn1.fyre.ibm.com:8443/gateway/wslpatch12-master-1/dsxhi/v1/delegationToken?doAs=user1&service=hdfs"
{"exception":null,"_statusCode_":500,"message":"
<MessageNotFound::org.apache.hadoop.security.token.SecretManager$InvalidToken: Renewal
request for unknown token\n\tat
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToke
n(AbstractDelegationTokenSecretManager.java:502)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewDelegationToken(FSNamesystem
.java:7171)\n\tat
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewDelegationToken(NameNo
deRpcServer.java:683)\n\tat
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewDeleg
ationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:1028)\n\tat
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtoco
l$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.j
ava:640)\n\tat
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)\n\tat
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)\n\tat
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)\n\tat
java.security.AccessController.doPrivileged(Native Method)\n\tat
javax.security.auth.Subject.doAs(Subject.java:422)\n\tat
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)\n\tat
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)\n>"}
Limitations
When you renew an HMS delegation token for HDP 2.x and CDH 5.x Hadoop clusters, for example:
curl -k -H "authorization: Bearer $TOKEN" -H "X-DelegationToken: $delegationToken" -X PUT "https://shadedgepn1.fyre.ibm.com:8443/gateway/wslpatch12-master-1/dsxhi/v1/delegationToken?doAs=user1&service=hms"
The following error message appears:
{"exception":null,"_statusCode_":500,"message":"<MessageNotFound::MetaException(message:dsxhi@FYRE.IBM.COM tries to renew a token with renewer dsxhi)>"}
This is a limitation of the Hive version in HDP 2.x and CDH 5.x. Hadoop clusters. This limitation was fixed in Hive versions 3.0.0 or later. For more information about Hive Jira, see: https://issues.apache.org/jira/browse/HIVE-16708:
Parent topic: Apache Hadoop