Topic
  • No replies
Artyom.Timofeev
Artyom.Timofeev
1 Post

Pinned topic GPFS remote copy and read operations are not working

‏2019-01-14T12:41:09Z |

Hello,

It would be nice if someone could help me with following problem:

 

I have an Ambari Cluster(2.7 version) with Spectrum Scale(4.2.x) installed. The service check for Spectrum Scale reports green. And i can browse the mounted gpfs. But some of the basic file operations do not work. Specifically i can't copy existing in gpfs files to another remote gpfs location and i cant download them to local filesystem. But i can upload new files to gpfs from local filesystem and i can execute -ls command. 

 

This is what i get when i do 'hadoop fs -copyToLocal':

 

WARN hdfs.DFSClient: Exception while reading from BP-hdfs,/gpfs/hdfs/hadoop/user/ven_atimofeev/.hiveJars/hive-exec-1.2.1000.2.6.5.0-292-d249a9484f801bbb96f01e7bbd357a58127aaca3e59c783a90c062bf99c9310d.jar:blk_965793_0 of /user/ven_atimofeev/.hiveJars/hive-exec-1.2.1000.2.6.5.0-292-d249a9484f801bbb96f01e7bbd357a58127aaca3e59c783a90c062bf99c9310d.jar from DatanodeInfoWithStorage[10.43.3.40:50010,DS-70d647db-ab5c-4b4a-ac12-ac0d07d7f5e7,DISK]

java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.43.3.40:59922 remote=/10.43.3.40:50010]

        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)

        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)

        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.readChannelFully(PacketReceiver.java:258)

        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:209)

        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)

        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:102)

        at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:201)

        at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:152)

        at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:781)

        at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:837)

        at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:897)

        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:945)

       at java.io.DataInputStream.read(DataInputStream.java:100)

        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:88)

        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:62)

        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:122)

        at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:467)

        at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:392)

        at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:329)

        at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:264)

        at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:249)

        at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)

        at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)

        at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:244)

        at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)

        at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)

        at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:221)

        at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:119)

        at org.apache.hadoop.fs.shell.Command.run(Command.java:165)

        at org.apache.hadoop.fs.FsShell.run(FsShell.java:297)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)

        at org.apache.hadoop.fs.FsShell.main(FsShell.java:356)

19/01/14 07:36:57 INFO hdfs.DFSClient: Could not obtain BP-hdfs,/gpfs/hdfs/hadoop/user/ven_atimofeev/.hiveJars/hive-exec-1.2.1000.2.6.5.0-292-d249a9484f801bbb96f01e7bbd357a58127aaca3e59c783a90c062bf99c9310d.jar:blk_965793_0 from any node: java.io.IOException: No live nodes contain block BP-hdfs,/gpfs/hdfs/hadoop/user/ven_atimofeev/.hiveJars/hive-exec-1.2.1000.2.6.5.0-292-d249a9484f801bbb96f01e7bbd357a58127aaca3e59c783a90c062bf99c9310d.jar:blk_965793_0 after checking nodes = [DatanodeInfoWithStorage[10.43.3.40:50010,DS-70d647db-ab5c-4b4a-ac12-ac0d07d7f5e7,DISK]], ignoredNodes = null No live nodes contain current block Block locations: DatanodeInfoWithStorage[10.43.3.40:50010,DS-70d647db-ab5c-4b4a-ac12-ac0d07d7f5e7,DISK] Dead nodes:  DatanodeInfoWithStorage[10.43.3.40:50010,DS-70d647db-ab5c-4b4a-ac12-ac0d07d7f5e7,DISK]. Will get new block locations from namenode and retry...

19/01/14 07:36:57 WARN hdfs.DFSClient: DFS chooseDataNode: got # 1 IOException, will wait for 2494.0127911234254 msec.

19/01/14 07:37:00 INFO hdfs.DFSClient: Could not obtain BP-hdfs,/gpfs/hdfs/hadoop/user/ven_atimofeev/.hiveJars/hive-exec-1.2.1000.2.6.5.0-292-d249a9484f801bbb96f01e7bbd357a58127aaca3e59c783a90c062bf99c9310d.jar:blk_965793_0 from any node: java.io.IOException: No live nodes contain block BP-hdfs,/gpfs/hdfs/hadoop/user/ven_atimofeev/.hiveJars/hive-exec-1.2.1000.2.6.5.0-292-d249a9484f801bbb96f01e7bbd357a58127aaca3e59c783a90c062bf99c9310d.jar:blk_965793_0 after checking nodes = [DatanodeInfoWithStorage[10.43.3.40:50010,DS-70d647db-ab5c-4b4a-ac12-ac0d07d7f5e7,DISK]], ignoredNodes = null No live nodes contain current block Block locations: DatanodeInfoWithStorage[10.43.3.40:50010,DS-70d647db-ab5c-4b4a-ac12-ac0d07d7f5e7,DISK] Dead nodes:  DatanodeInfoWithStorage[10.43.3.40:50010,DS-70d647db-ab5c-4b4a-ac12-ac0d07d7f5e7,DISK]. Will get new block locations from namenode and retry...