Troubleshooting
Problem
You have configured the Hadoop stage to use Hadoop config files but SDC can’t find them in the SDC_RESOURCES directory under the directory name you configured on the pipeline.
Symptom
You are using a Hadoop or Hive stage and are getting a validation error like this one when trying to run/preview/validate the pipeline:
Validation issue: HADOOPFS_25 - Hadoop configuration directory '/var/lib/sdc-resources/<Configuration Files Directory>' does not exist
The path in quotes can change depending on your SDC_RESOURCES directory and the value of the Configuration Files Directory property in your pipeline.
Resolving The Problem
SDC will look up these files at the path $SDC_RESOURCES/<Configuration Files Directory property>. As the error message suggests this error can be due to the following:
The HDFS config files (core-site.xml and hdfs-site.xml for Hadoop stages) or symlink pointing to the are not present in SDC_RESOURCES.
The symlink to the config files is broken.
To fix this make sure the config files are accessible to Data Collector by copying or creating a symlink to them in the SDC_RESOURCES directory. If you already set up a symlink, ensure that the files exist.
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
15 March 2025
UID
ibm17186175