PM99046: HADOOP FAILED TO START DUE TO NAMENODE IS STILL IN SAFEMODE.

APAR status

• Closed as program error.

Error description

• With more disks for the datanodes, after upgrade to BI 2.1.0.1,
  start.sh hadoop may fail with following errors:
  
  [ERROR] @<hostname> - tasktracker failed to start because
  namenode is still in safemode probably because of too many
  blocks missing. Please check namenode log file under
  /var/ibm/biginsights/hadoop/logs on <hostname>
  [INFO] Progress - 19%
  [ERROR] @<hostname> - tasktracker failed to start because
  namenode is still in safemode probably because of too many
  blocks missing. Please check namenode log file under
  /var/ibm/biginsights/hadoop/logs on <hostname>
  [INFO] Progress - 25%
  [ERROR] @<hostname> - tasktracker failed to start because
  namenode is still in safemode probably because of too many
  blocks missing. Please check namenode log file under
  /var/ibm/biginsights/hadoop/logs on <hostname>
  [INFO] Progress - 31%
  [ERROR] @<hostname> - tasktracker failed to start because
  namenode is still in safemode probably because of too many
  blocks missing. Please check namenode log file under
  /var/ibm/biginsights/hadoop/logs on <hostname>
  [INFO] Progress - 38%
  [ERROR] @<hostname> - tasktracker failed to start because
  namenode is still in safemode probably because of too many
  blocks missing. Please check namenode log file under
  /var/ibm/biginsights/hadoop/logs on <hostname>
  [INFO] Progress - 44%
  [ERROR] @<hostname> - tasktracker failed to start because
  namenode is still in safemode probably because of too many
  blocks missing. Please check namenode log file under
  /var/ibm/biginsights/hadoop/logs on <hostname>
  [INFO] Progress - 50%
  [INFO] Progress - 100%
  [ERROR] DeployManager -
  [ERROR] DeployManager - OPERATION ERROR -- Start [hadoop]:
  [ERROR] DeployManager -
  -------------------------------------------------------
  [ERROR] DeployManager - hadoop failed
  com.ibm.xap.mgmt.DeployException: HDFS failed to start because
  number of available datanodes:[0] is less than
  dfs.replication.min:[1]
          at
  com.ibm.xap.mgmt.hdm.HadoopDeployer.startService(HadoopDeployer.
  java:455)
          at
  com.ibm.xap.mgmt.DeployManager$StartThread.work(DeployManager.ja
  va:2595)
          at
  com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.ja
  va:2387)
  [ERROR] DeployManager - Start; SUCCEEDED components: []; FAILED
  components: [hadoop]; Consumes : 44276ms
  Error exit.
  
  Defect 51920 has created to track this issue.
  

Local fix

• run start.sh hadoop twice (once to start HDFS, and once to
  start mapred which fails the first time).
  

Problem summary

• See error description
  

Problem conclusion

• This problem is fixed in Version 2.1.0.2 and later fix packs.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  INFO BIGINSIGHT

• Fixed component ID

  5725C0900

Applicable component levels

• R212 PSY

     UP