映射器代码使用unix管道运行，但不使用hadoop流。错误NA。流命令失败

Question

我正在尝试在Hadoop流中解决倒置单词列表问题（对于每个单词，输出是包含该单词的文件名列表）。输入是包含文本文件的目录的名称。我已经用python编写了映射器和化简器，它们在尝试使用unix管道时工作正常。但是，当使用Hadoop流命令执行时，代码会运行，但作业最终会失败。我怀疑这是Mapper代码中的东西，但似乎无法确切知道问题所在。

我是一个初学者（如果我没有得到正确的东西，请原谅），在VMware Fusion上使用Cloudera培训。我将Mapper和Reducer.py可执行文件放在本地系统和hdfs的主目录中。我有hdfs上的目录“莎士比亚”。下面的unix管道命令工作正常。

回声莎士比亚|/反相器。py|排序|./InvertedReducer.py

然而，haddop流并没有。

hadoop jar /usr/lib/hadoop-0.20-映射减量/贡献/流/hadoop-流*.jar -输入莎士比亚 -输出反转列表 -映射器 InvertedMapper.py -化简器 InvertedReducer.py -文件 InvertedMapper.py -文件 InvertedReducer.py

#MAPPER CODE

#!/usr/bin/env python

import sys
import os

class Mapper(object):

        def __init__(self, stream, sep='\t'):
                self.stream=stream
                self.sep=sep

        def __iter__(self):
                os.chdir(self.stream.read().strip())
                files = [os.path.abspath(f) for f in os.listdir(".")]
                for file in files:
                        yield file

        def emit(self, key, value):
                sys.stdout.write("{0}{1}{2}\n".format(key,self.sep,value))

        def map(self):
                for file in self:
                        with open(file) as infile:
                                name = file.split("/")[-1].split(".")[0]
                                words = infile.read().strip().split()
                                for word in words:
                                        self.emit(word,name)

 if __name__ == "__main__":
        cwd = os.getcwd()
        mapper = Mapper(sys.stdin)
        mapper.map()
        os.chdir(cwd)


#REDUCER CODE

#!/usr/bin/env python

import sys
from itertools import groupby
from operator import itemgetter

class Reducer(object):
        def __init__(self, stream, sep="\t"):
                self.stream = stream
                self.sep = sep

        def __iter__(self):
                for line in self.stream:
                        try:
                                parts = line.strip().split(self.sep)
                                yield parts[0], parts[1]
                        except:
                                continue

        def emit(self, key, value):
                sys.stdout.write("{0}{1}{2}\n".format(key, self.sep, value))

        def reduce(self):
                for key, group in groupby(self, itemgetter(0)):
                        values = []
                        for item in group:
                                values.append(item[1])
                        values = set(values)
                        values = list(values)
                        self.emit(key, values)
if __name__ == "__main__":
    reducer = Reducer(sys.stdin)
    reducer.reduce()

运行Hadoop命令的输出如下。

packageJobJar: [InvertedMapper1.py, /tmp/hadoop-training/hadoop-unjar281431668511629942/] [] /tmp/streamjob679048425003800890.jar tmpDir=null
19/02/17 00:22:19 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
19/02/17 00:22:19 INFO mapred.FileInputFormat: Total input paths to process : 5
19/02/17 00:22:20 INFO streaming.StreamJob: getLocalDirs(): [/var/lib/hadoop-hdfs/cache/training/mapred/local]
19/02/17 00:22:20 INFO streaming.StreamJob: Running job: job_201902041621_0051
19/02/17 00:22:20 INFO streaming.StreamJob: To kill this job, run:
19/02/17 00:22:20 INFO streaming.StreamJob: UNDEF/bin/hadoop job  -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201902041621_0051
19/02/17 00:22:20 INFO streaming.StreamJob: Tracking URL: http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201902041621_0051
19/02/17 00:22:21 INFO streaming.StreamJob:  map 0%  reduce 0%
19/02/17 00:22:34 INFO streaming.StreamJob:  map 40%  reduce 0%
19/02/17 00:22:39 INFO streaming.StreamJob:  map 0%  reduce 0%
19/02/17 00:22:50 INFO streaming.StreamJob:  map 40%  reduce 0%
19/02/17 00:22:53 INFO streaming.StreamJob:  map 0%  reduce 0%
19/02/17 00:23:03 INFO streaming.StreamJob:  map 40%  reduce 0%
19/02/17 00:23:06 INFO streaming.StreamJob:  map 20%  reduce 0%
19/02/17 00:23:07 INFO streaming.StreamJob:  map 0%  reduce 0%
19/02/17 00:23:16 INFO streaming.StreamJob:  map 20%  reduce 0%
19/02/17 00:23:17 INFO streaming.StreamJob:  map 40%  reduce 0%
19/02/17 00:23:19 INFO streaming.StreamJob:  map 20%  reduce 0%
19/02/17 00:23:21 INFO streaming.StreamJob:  map 100%  reduce 100%
19/02/17 00:23:21 INFO streaming.StreamJob: To kill this job, run:
19/02/17 00:23:21 INFO streaming.StreamJob: UNDEF/bin/hadoop job  -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201902041621_0051
19/02/17 00:23:21 INFO streaming.StreamJob: Tracking URL: http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201902041621_0051
19/02/17 00:23:21 ERROR streaming.StreamJob: Job not successful. Error: NA
19/02/17 00:23:21 INFO streaming.StreamJob: killJob...
Streaming Command Failed!

Answer 1

我不知道这是不是你的代码失败的原因，但是FAQ指出不应该在Hadoop Streaming中使用unix管道。

https://Hadoop . Apache . org/docs/current/Hadoop-streaming/Hadoop streaming . html #常见问题