IBM Support

java.io.IOException: No FileSystem for scheme: gpfs

Troubleshooting


Problem

The user installed Conductor with Spark in Symphony cluster and the underground file system for data storage is IBM Spectrum Scale(GPFS). When Hive users ran Spark SQL jobs via ThriftServer, jobs failed with below error messages. How to submit jobs: From the client node, $beeline -u jdbc:hive2://xxxxxx:10015 -n xxxxxx $select * from tdl_cm_cell limit 10;

Symptom

Error messages after run select query:
18/01/19 15:12:51 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 305.1 KB, free 5.8 GB)
18/01/19 15:12:51 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 24.9 KB, free 5.8 GB)
18/01/19 15:12:51 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.48.134.87:44377 (size: 24.9 KB, free: 5.8 GB)
18/01/19 15:12:51 INFO SparkContext: Created broadcast 2 from run at AccessController.java:686
18/01/19 15:12:51 ERROR SparkExecuteStatementOperation: Error executing query, currentState RUNNING,
java.io.IOException: No FileSystem for scheme: gpfs
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
        ......
......
      ......
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(Thread.java:785)
18/01/19 15:12:51 ERROR SparkExecuteStatementOperation: Error running hive query:
org.apache.hive.service.cli.HiveSQLException: java.io.IOException: No FileSystem for scheme: gpfs
        at
......
......

[{"Product":{"code":"SSZU2E","label":"IBM Spectrum Conductor with Spark"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Component":"--","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"2.2","Edition":"","Line of Business":{"code":"LOB77","label":"Automation Platform"}}]

Log InLog in to view more of this document

This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.

Document Information

Modified date:
17 June 2018

UID

isg3T1026956

Page Feedback