Command line for HDFS Transparency

Edit online

The HDFS shell command line can be used with HDFS Transparency.

Table 1. HDFS shell command line for HDFS Transparency
Command interface Sub-commands Comments
hdfs dfs -xxx

hadoop dfs -xxx

 Supported hdfs dfs -du does not report exact total value for a directory for all the HDFS Transparency versions before HDFS Transparency 3.1.0-1.
Note: For HDFS Transparency, hdfs dfs -du /path/to/target needs to recursively go through all directories and files under <gpfs.mnt.dir>/<gpfs.data.dir>/path/to/target. If you have a lot of subdirectories and files under <gpfs.mnt.dir>/<gpfs.data.dir>/path/to/target, it is recommended to not run this command frequently.
Note:
  • For HDFS Transparency, hdfs dfs -du /path/to/target needs to recursively go through all the directories and files under <gpfs.mnt.dir>/<gpfs.data.dir>/path/to/target. If there are many subdirectories and files under <gpfs.mnt.dir>/<gpfs.data.dir>/path/to/target, it is recommended to not run this command frequently.
  • If you are using the Hadoop dfs -du command to get the output size of a file, the size value might not correspond to the du command output from the same file on the IBM Storage® Scale file system. The IBM Storage Scale file system will consider replication factor and snapshot value into the file count value. Therefore, use the POSIX ls -l output file size on the IBM Storage Scale file system to compare with the Hadoop du command output file size.
hdfs envvars Supported
hdfs getconf Supported
hdfs groups Supported
hdfs jmxget Supported
hdfs haadmin Supported
hdfs zkfc Supported
hdfs crypto Supported since HDFS Transparency 3.0.0+ CES HDFS does not support hdfs crypto -listZones. For a workaround with the GPFS policy engine, see the step 31 in the Second generation HDFS Transparency Protocol troubleshooting topic.
hdfs httpfs Not tested Take HDFS WebHDFS for REST API
hdfs lsSnapshottableDir Not supported
hdfs oev Not supported
hdfs oiv Not supported
hdfs oiv_legacy Not supported
hdfs snapshotDiff Not supported
hdfs balancer Not supported
hdfs cacheadmin Not supported
hdfs diskbalancer Not supported
hdfs ec Not supported
hdfs journalnode Not supported
hdfs mover Not supported
hdfs namenode Not supported
hdfs nfs3 Not supported
hdfs portmap Not supported
hdfs secondarynamenode Not supported
hdfs storagepolicies Not supported
hdfs dfsadmin Not supported
Note: HDFS administrative commands, such as hdfs haadmin and hdfs groups cannot be executed from HDFS clients where Kerberos is enabled. The HDFS client ensures that the CES-HDFS user principle has the CES-HOST name instead of the NameNode hostname. The administrative commands fail while doing the hostname matching.
To resolve this, we have to add the following key in the core-site.xml file on the client: 
hadoop.security.service.user.name.key.pattern=*