Hadoop：所有数据阳极127.0.0.1:50010都是坏的。正在中止

Question

14/12/10 10:58:36 INFO mapred.JobClient: Running job: job_201412091528_0004
14/12/10 10:58:37 INFO mapred.JobClient:  map 0% reduce 0%
14/12/10 10:58:50 INFO mapred.JobClient:  map 10% reduce 0%
14/12/10 10:59:44 INFO mapred.JobClient:  map 20% reduce 0%
14/12/10 11:32:23 INFO mapred.JobClient: Task Id : attempt_201412091528_0004_m_000000_0, Status : FAILED
java.io.IOException: All datanodes 127.0.0.1:50010 are bad. Aborting...
  at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3290)
  at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2783)
  at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2987)

2014-12-10 11:32:19,157 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:50010, dest: /127.0.0.1:62662, bytes: 549024, op: HDFS_READ, cliID: DFSClient_attempt_201412091528_0004_m_000000_0_1249767243_1, offset: 0, srvID: DS-957555695-10.0.1.9-50010-1418164764736, blockid: blk_-7548306051547444322_12804, duration: 2012297405000
2014-12-10 11:32:25,511 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:50010, dest: /127.0.0.1:64109, bytes: 1329, op: HDFS_READ, cliID: DFSClient_attempt_201412091528_0004_m_000000_1_-1362169490_1, offset: 0, srvID: DS-957555695-10.0.1.9-50010-1418164764736, blockid: blk_4126779969492093101_12817, duration: 285000
2014-12-10 11:32:29,496 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:50010, dest: /127.0.0.1:64110, bytes: 67633152, op: HDFS_READ, cliID: DFSClient_attempt_201412091528_0004_m_000000_1_-1362169490_1, offset: 0, srvID: DS-9575556

...或在namenode文件中。jobtracker只是重复在datanode日志中发现的错误。在失败之前几分钟的一个错误是EOF错误,对于PartialBuilder来说,这可能是问题,也可能不是问题：

2014-12-10 12:12:22,060 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-957555695-10.0.1.9-50010-1418164764736, infoPort=50075, ipcPort=50020):DataXceiver
java.io.EOFException: while trying to read 65557 bytes
  at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:296)
  at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:340)
  at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:404)
  at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:582)
  at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:404)
  at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
  at java.lang.Thread.run(Thread.java:695)

我能够直接读写文件到DFS。我甚至可以在数据的一个很小的子集上运行这个作业，但是我不能让这个map/reduce作业正确工作。知道我做错了什么吗？

有关我得安装得说明：

null

• dfs.replication=4
• dfs.permissions=false
• dfs.name.dir=/users/juliuss/dev/hdfs/name/
• dfs.data.dir=/users/juliuss/dev/hdfs/data/
• mapred.child.java.opts=-xmx4096m
• dfs.datanode.socket.write.timeout=0

Answer 1

在折腾了一百万个设置之后，没有一个有效，我最终通过大幅减小拆分大小来解决这个问题：

-Dmapred.max.split.size=16777216

这将此数据集的映射器数量从10个增加到40个，从而允许它们正确完成。现在我已经隔离了这个问题，我将稳步增加拆分大小以找到正确的数字。（对于随机森林，您应该找到尽可能大的分裂，以获得最好的结果。）

不幸的是，我不知道为什么拆分的大小会导致“所有数据阳极都坏了。中止”错误，因为这不是我想要的错误。