当前位置: 首页 > 工具软件 > OSChina Block > 使用案例 >

运行wordcount时显示Could not obtain block

宿建本
2023-12-01
该文章接上面hadoop运行wordcount时卡住不动,接着下面
hadoop@ubuntu118:~/hadoop-1.0.2$ bin/hadoop dfsadmin -safemode leave
Warning: $HADOOP_HOME is deprecated.

Safe mode is OFF

hadoop@ubuntu118:~/hadoop-1.0.2$ bin/hadoop jar hadoop-examples-1.0.2.jar wordcount file fileout66
Warning: $HADOOP_HOME is deprecated.

d****hdfs://192.168.0.118:9000/user/hadoop/file
12/09/10 18:35:52 INFO input.FileInputFormat: Total input paths to process : 2
12/09/10 18:35:53 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/09/10 18:35:53 WARN snappy.LoadSnappy: Snappy native library not loaded
12/09/10 18:35:53 INFO mapred.JobClient: Running job: job_201209101835_0001
12/09/10 18:35:54 INFO mapred.JobClient:  map 0% reduce 0%
12/09/10 18:36:17 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000000_0, Status : FAILED
java.io.IOException: Could not obtain block: blk_-7132067374412134968_1005 file=/user/hadoop/file/file2
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
        at java.io.DataInputStream.read(DataInputStream.java:83)
        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
        at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/09/10 18:36:20 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000001_0, Status : FAILED
java.io.IOException: Could not obtain block: blk_-4697097477082743821_1006 file=/user/hadoop/file/file1
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
        at java.io.DataInputStream.read(DataInputStream.java:83)
        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
        at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/09/10 18:36:35 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000000_1, Status : FAILED
java.io.IOException: Could not obtain block: blk_-7132067374412134968_1005 file=/user/hadoop/file/file2
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
        at java.io.DataInputStream.read(DataInputStream.java:83)
        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
        at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/09/10 18:36:35 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000001_1, Status : FAILED
java.io.IOException: Could not obtain block: blk_-4697097477082743821_1006 file=/user/hadoop/file/file1
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
        at java.io.DataInputStream.read(DataInputStream.java:83)
        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
        at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/09/10 18:36:50 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000000_2, Status : FAILED
java.io.IOException: Could not obtain block: blk_-7132067374412134968_1005 file=/user/hadoop/file/file2
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
        at java.io.DataInputStream.read(DataInputStream.java:83)
        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
        at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/09/10 18:36:50 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000001_2, Status : FAILED
java.io.IOException: Could not obtain block: blk_-4697097477082743821_1006 file=/user/hadoop/file/file1
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
        at java.io.DataInputStream.read(DataInputStream.java:83)
        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
        at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/09/10 18:37:12 INFO mapred.JobClient: Job complete: job_201209101835_0001
12/09/10 18:37:12 INFO mapred.JobClient: Counters: 6
12/09/10 18:37:12 INFO mapred.JobClient:   Job Counters 
12/09/10 18:37:12 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=115005
12/09/10 18:37:12 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/09/10 18:37:12 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/09/10 18:37:12 INFO mapred.JobClient:     Launched map tasks=8
12/09/10 18:37:12 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
12/09/10 18:37:12 INFO mapred.JobClient:     Failed map tasks=1

执行文件file里面文件权限问题,查看里面的文件后发现其中的文件没有执行权限,但修改后发现还是没有执行权限。无奈执行重新上传文件上去hdfs,之后运行正常。

转载于:https://my.oschina.net/mkh/blog/77852

 类似资料: