hadoop mapreduce作业不运行reducer

Question

我试图运行WordCount示例的一个变体，这个变体是，映射器输出文本作为键和文本作为值，而还原器输出文本作为键和NullWritable作为值。

除了地图，减少签名，我把主要的方法是这样的：

//start a conf
Configuration conf = new Configuration();
conf.set("str",str);

//initialize a job based on the conf
Job job = new Job(conf, "wordcount");
job.setJarByClass(org.myorg.WordCount.class);

//the reduce output
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(NullWritable.class);

//the map output
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);

//Map and Reduce
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);


//take hdfs locations as input and output
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);

FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

//run the job
job.waitForCompletion(true);

map(LongWritable key, Text value, Context context){
.........
context.write("1000000","2");
}

reduce(Text key, Iterable<Text> values, Context context){
 .......
 context.write("v",NullWritable.get());
}

Answer 1

尝试将@override添加到reduce函数中，以确保它实际上覆盖了该行为。如果您的签名不匹配，那么它将使用默认的reduce，它不执行任何操作。如果没有正确重写，将会出现编译器错误。

@override
reduce(Text key, Iterable<Text> values, Context context){
 .......
 context.write("v",NullWritable.get());
}