StreamSets Manual Inputs

By default, Manta Flow extracts StreamSets pipelines from the configured StreamSets Data Collector (SDC) server and then analyzes the data flows in them. It is also possible to analyze the data flows in the manually-provided StreamSets pipelines exported from SDC.

Input Folder Structure

To analyze SDC pipelines and jobs, create a directory structure as follows.

• temp/streamsets/${streamsets.extractor.server}
  • pipelines/ directory with pipeline files (optional, used only if pipelines are used)
    • *.json all exported pipelines
  • jobs/ directory with job files (optional, used only if jobs are used)
    • *.json all exported jobs
  • connections_metadata.json
  • dataCollectors.csv - see StreamSets Resource Configuration:Data Collector Settings for more details