IJ29106: HDFS TRANSPARENCY NAMENODE NULLPOINTEREXCEPTIONS

APAR status

• Closed as program error.

Error description

• The Spectrum Scale HDFS Transparency connector version
  3.1.0-6 contains 2 NullPointerExceptions in the
  HDFS NameNode service. The application accessing the
  data is not impacted, but these exceptions are seen
  in the NameNode log file.
  
  The following are examples of each exception.
  
  Exception 1: getBlockLocations
  
  2020-10-25 18:37:09,484 WARN  ipc.Server
  (Server.java:logException(2717)) - IPC Server handler 457
  on 8020, call Call#369 Retry#0
  org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLo
  cations from 10.127.64.144:43109
  java.lang.NullPointerException
          at
  org.apache.hadoop.hdfs.server.blockmanagement.GPFSBlockMa
  nager.getStorages(GPFSBlockManager.java:102)
          at
  org.apache.hadoop.hdfs.server.blockmanagement.GPFSBlockMa
  nager.createLocatedBlock(GPFSBlockManager.java:237)
          at
  org.apache.hadoop.hdfs.server.blockmanagement.GPFSBlockMa
  nager.createLocatedBlockList(GPFSBlockManager.java:178)
          at
  org.apache.hadoop.hdfs.server.blockmanagement.GPFSBlockMa
  nager.createLocatedBlocks(GPFSBlockManager.java:151)
          at
  org.apache.hadoop.hdfs.server.blockmanagement.GPFSBlockMa
  nager.createLocatedBlocks(GPFSBlockManager.java:130)
          at
  org.apache.hadoop.hdfs.server.namenode.GPFSNamesystemV0.g
  etBlockLocations(GPFSNamesystemV0.java:347)
          at
  org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
  getBlockLocations(NameNodeRpcServer.java:727)
          at
  org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolS
  erverSideTranslatorPB.getBlockLocations(ClientNamenodePro
  tocolServerSideTranslatorPB.java:414)
          at
  org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProto
  colProtos$ClientNamenodeProtocol$2.callBlockingMethod(Cli
  entNamenodeProtocolProtos.java)
          at
  org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRp
  cInvoker.call(ProtobufRpcEngine.java:523)
          at
  org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
          at
  org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
          at
  org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
          at
  java.security.AccessController.doPrivileged(Native
  Method)
          at
  javax.security.auth.Subject.doAs(Subject.java:422)
          at
  org.apache.hadoop.security.UserGroupInformation.doAs(User
  GroupInformation.java:1682)
          at
  org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675
  )
  
  Exception 2: mkdirs
  
  2020-10-25 18:02:49,611 WARN  ipc.Server
  (Server.java:logException(2717)) - IPC Server handler
  1288 on 8020, call Call#569 Retry#0
  org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs
  from 10.127.64.153:25906
  java.lang.NullPointerException
          at
  org.apache.hadoop.hdfs.server.namenode.GPFSNamesystemV0.c
  reateDirectories(GPFSNamesystemV0.java:2877)
          at
  org.apache.hadoop.hdfs.server.namenode.GPFSNamesystemV0.m
  kdirs(GPFSNamesystemV0.java:1481)
          at
  org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
  mkdirs(NameNodeRpcServer.java:1115)
          at
  org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolS
  erverSideTranslatorPB.mkdirs(ClientNamenodeProtocolServer
  SideTranslatorPB.java:695)
          at
  org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProto
  colProtos$ClientNamenodeProtocol$2.callBlockingMethod(Cli
  entNamenodeProtocolProtos.java)
          at
  org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRp
  cInvoker.call(ProtobufRpcEngine.java:523)
          at
  org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
          at
  org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
          at
  org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
          at
  java.security.AccessController.doPrivileged(Native
  Method)
          at
  javax.security.auth.Subject.doAs(Subject.java:422)
          at
  org.apache.hadoop.security.UserGroupInformation.doAs(User
  GroupInformation.java:1682)
          at
  org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675
  )
  
  
  The problem was reported on HDFS Transparency
  version 3.1.0-6 with HDP 3.1.4.41 on x86_64 Linux.
  
  
  Known Impact:
  
  The NullPointerExceptions do not cause any outage of
  the HDFS service nor do they cause applicaitons to
  fail to read or write to the HDFS file system.
  

Local fix

Problem summary

• The Spectrum Scale HDFS Transparency connector version
  3.1.0-6 contains 2 NullPointerExceptions in the HDFS
  NameNode service. The application accessing the data
  is not impacted, but these exceptions are seen in the
  NameNode log file
  

Problem conclusion

• Benefits of the solution:
  Fixed the code so there are no longer any
  NullPointerExceptions in the HDFS Transparency
  NameNode log file.
  * Work around: None
  * Problem trigger: User running HDFS Transparency workload.
  * Symptom: Error output/message.
  * Platforms affected: All
  * Functional Area affected: HDFS Connector.
  * Customer Impact: High
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  IJ30337

Fix information

• Fixed component name

  SPEC SCALE DME

• Fixed component ID

  5737F34AP

Applicable component levels