wordcount程序中的hadoop异常类型不匹配

Question

我是新手，尝试运行我第一个hadoop程序。我在Hadoop中执行wordcount作业时遇到了一些问题。

package hdp;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.util.Tool; 
import org.apache.hadoop.util.ToolRunner;

public class WordCount extends Configured implements Tool{
    public static void main(String[] args) throws Exception {
        System.out.println("application starting ....");
        int exitCode = ToolRunner.run(new WordCount(), args);
        System.out.println(exitCode);
    }

    @Override
    public int run(String[] args) throws Exception {
        if (args.length < 2) {
            System.out.println("Plz enter input and output directory properly... ");
            return -1;
        }
        JobConf conf = new JobConf(WordCount.class);
        FileInputFormat.setInputPaths(conf, new Path(args[0]));
        FileOutputFormat.setOutputPath(conf, new Path(args[1]));

        conf.setMapperClass(WordMapper.class);
        conf.setReducerClass(WordReducer.class);

        conf.setMapOutputKeyClass(Text.class);
        conf.setMapOutputKeyClass(IntWritable.class);

        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(IntWritable.class);

        JobClient.runJob(conf);
        return 0;
    }

    @Override
    public Configuration getConf() {
        return null;
    }

    @Override
    public void setConf(Configuration arg0) {

    }   
}

import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

public class WordMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable>{

    @Override
    public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> collect, Reporter reporter) throws IOException {
        String str = value.toString();
        for (String s : str.split(" ")) {
            if (s.length() > 0) {
                collect.collect(new Text(s), new IntWritable(1));
            }
        }
    }
}

package hdp;
import java.io.IOException;
import java.util.Iterator; 
import org.apache.hadoop.io.IntWritable; 
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase; 
import org.apache.hadoop.mapred.OutputCollector; 
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;

public class WordReducer extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable>  {

    @Override
    public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
        int count = 0;
        while (values.hasNext()) {
            IntWritable intWritable = values.next();
            count += intWritable.get();
        }
        output.collect(key, new IntWritable(count));
    }
}

当我运行我的程序，然后我得到以下错误消息。

16/12/23 00:22:41信息MapReduce.job：任务Id：尝试_1482432671993_0001_M_000001_1,状态：失败错误：java.io.ioException：映射项中的类型不匹配：应为org.apache.hadoop.io.intwritable，已收到org.apache.hadoop.mapred.maptask$MapOutputBuffer.Collect(MapTask.java:1072)在org.apache.hadoop.mapred.maptask$OldOutputCollector.Collect16/12/23 00:22:47信息MapReduce.job：任务Id：尝试_1482432671993_0001_M_000000_2,状态：失败错误：java.io.ioException：映射中的键类型不匹配：应为org.apache.hadoop.io.intwritable，已收到org.apache.hadoop.mapred.maptask$MapOutputBuffer.Collect(MapTask.java:1072)在org.apache.hadoop.japtask$OldOutputCollector.Collect(MapTask.java:610)在

请告诉我我错在哪里，我需要什么样的改变。在WordCount.java或WordReducer或WordMapper.java中

Answer 1

您意外地设置了输出键类两次：

conf.setMapOutputKeyClass(IntWritable.class);

应成为

conf.setMapOutputValueClass(IntWritable.class);