Functional limitations

This topic lists the functional limitations.

General

• The maximum number of Extended Attributes (EA) is limited by IBM Spectrum® Scale. The total size of the EA key and value must be less than a metadata block size in IBM Spectrum Scale.
• The EA operation on snapshots is not supported.
• Raw namespace is not implemented because it is not used internally.
• If gpfs.replica.enforced is configured as gpfs, the Hadoop shell command hadoop dfs -setrep does not take effect. Also, hadoop dfs -setrep -w stops functioning and does not exit.
• HDFS Transparency NameNode does not provide safemode because it is stateless.
• HDFS Transparency NameNode does not need the second NameNode like native HDFS because it is stateless.
• Maximal replica for IBM Spectrum Scale is 3 from code. However, the maximal replica for your file system might be less than 3. You can check this by running /usr/lpp/mmfs/bin/mmlsfs <fs-name> -R.
• IBM Spectrum Scale has no ACL entry number limit. The maximal entry number is limited by Int32.
• SendPacketDownStreamAvgInfo and SlowPeersReport from http://<namenode/datanode:port>/jmx are not supported.
• GPFS™ file data replication factor on ESS requires to be set to 1, and dfs.replica should be set to 1.
• HDFS supported interface for hdfs xxx is hdfs dfs xxx. Other interface from hdfs xxx is considered native HDFS specific, that is not used by the HDFS Transparency.
  These are some examples of what is not supported:

  • fsck
  • dfsadmin
    • - safemode
  • Native HDFS caching (cacheadmin)
  • NameNode format not needed to run (namenode -format)
• Distcp over snapshot is not supported.
• For HDFS Transparency version 3.0.x, the environment variables above can be exported, except for HADOOP_COMMON_LIB_NATIVE_DIR.

  This is because HDFS Transparency uses its own native .so library.

  For HDFS Transparency version 3.0.x:

  • If you did not export HADOOP_CONF_DIR, then HDFS Transparency will read all the configuration files under /var/mmfs/hadoop/etc/hadoop such as the gpfs-site.xml file and the hadoop-env.sh file.
  • If you export HADOOP_CONF_DIR, then HDFS Transparency will read all the configuration files under $HADOOP_CONF_DIR. Since gpfs-site.xml is required for HDFS Transparency, it will only read the gpfs-site.xml file from the /var/mmfs/hadoop/etc/hadoop directory.