Short circuit write

Short circuit write is supported since HDFS Transparency 2.7.3-1.

If HDFS client and HDFS Transparency DataNode are located on the same node, when writing file from HDFS client, short circuit write will write data directly into IBM Storage Scale file system instead of writing data through RPC. This could reduce the RPC latency through the local loop network adapter and thus enhance the write performance.

Figure 1. Short Circuit Write Logic
Short Circuit Write Logic

In Figure 1, (A) is for the original logic for data write. With short circuit write enabled, the data write logic will be shown as (B). The data will be written directly into IBM Storage Scale file system.

If you want to enable this feature, refer Short-circuit read configuration to enable short circuit read first. By default, when short circuit read is enabled, short circuit write is also enabled. When short circuit read is disabled, short circuit write is also disabled.

If you want to disable short circuit write when short circuit read is enabled:
  1. Add the following configuration in hdfs-site.xml for Hadoop client.
    If you take HortonWorks HDP, change this on Ambari/HDFS/configs and restart HDFS service. If you take open source Apache Hadoop, change this in <Hadoop-home-dir>/etc/hadoop/hdfs-site.xml on all Hadoop nodes.
    <property>
<name>gpfs.short-circuit-write.enabled</name>
<value>false</value>
</property>
  2. Add the same configuration into gpfs-site.xml.

    If you take HortonWorks HDP, change this on Ambari/Spectrum Scale/Configs/custom gpfs-site and restart IBM Storage Scale service from Ambari.

    If you take open source Apache Hadoop, change this in /usr/lpp/mmfs/hadoop/etc/hadoop/gpfs-site.xml (for HDFS Transparency 2.7.3-x) or /var/mmfs/hadoop/etc/hadoop/gpfs-site.xml (for HDFS Transparency 3.0.x) and run /usr/lpp/mmfs/bin/mmhadoopctl connector syncconf /usr/lpp/mmfs/hadoop/etc/hadoop (for HDFS Transparency 2.7.3-x) or /usr/lpp/mmfs/bin/mmhadoopctl connector syncconf /var/mmfs/hadoop/etc/hadoop (for HDFS Transparency 3.0.x) to sync the change to all HDFS Transparency nodes.