MapReduce减速器的键出类型

Question

我写了映射和Reduce程序，其中reducer的输出键和值不同于它的输入或映射器的输出。我在司机班上做了适当的改变。下面是我在运行它时得到的异常：

信息MapReduce.job：任务Id:Attribut_1550670375771_4211_M_0000032，状态：失败错误：java.io.ioException：map中的值类型不匹配：expected org.apache.hadoop.io.text,received org.apache.hadoop.maptask$MapOutputBuffer.Collect(maptask.java:1084).在org.apache.hadoop.mapreduce.lib.map.wrappedmapper$context.write(wrappdmapper.java:112)在com.hirw.maxClosePrice.myHadoopmapper.map（myHadoopmapper.java:20)在com.hirw.maxClosePrice.myHadoopmapper.map（myHadoopmapper.java:1)在org.apache.hadoop.maprede.mapper.run（mapper.java:145)在runnewMapper(maptask.java:793)，地址为org.apache.hadoop.mapred.maptask.run(maptask.java:341)，地址为org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:164)，地址为java.security.accessController.doPrivileged（本机方法），地址为javax.security.auth.subject.doas(Subject.java:415)，地址为org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1920)，地址为org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:158)

19/04/16 22:24:50 INFO MapReduce.Job:map 100%reduce 100%19/04/16 22:24:50 INFO MapReduce.Job:job_1550670375771_4211失败，状态失败，原因是：任务失败task_1550670375771_4211_m_000001任务失败时作业失败。FailedMaps：1 FailedReduces：0

当Reducer的KeyOut和ValueOut与Mapper的KeyOut和ValueOut相同时，它可以工作，但当它们不同时，它就失效了。

我的映射器类：公共类MyHadoopMapper扩展映射器{

@Override
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

    String[] recordItems = value.toString().split(",");

    String stock = recordItems[1];
    Float stockValue = Float.parseFloat(recordItems[6]);

    context.write(new Text(stock), new FloatWritable(stockValue));
}

}

减速器类：

公共类MyHadoopReducer扩展Reducer{

@Override
public void reduce(Text key, Iterable<FloatWritable> values, Context context
        ) throws IOException, InterruptedException {

    Float maxVal = Float.MIN_VALUE;
    for (FloatWritable stockValue : values) {
        maxVal = stockValue.get() > maxVal ? stockValue.get() : maxVal;
    }

    context.write(key, new Text(String.valueOf(maxVal)));
}

}

驱动程序类：公共类MyHadoopDriver{

public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
    // TODO Auto-generated method stub

    Job hadoopJob = new Job();
    hadoopJob.setJarByClass(MyHadoopDriver.class);
    hadoopJob.setJobName("MyStockPrice");

    FileInputFormat.addInputPath(hadoopJob, new Path("/user/hirw/input/stocks"));
    FileOutputFormat.setOutputPath(hadoopJob, new Path("stocksData"));

    hadoopJob.setInputFormatClass(TextInputFormat.class);
    hadoopJob.setOutputFormatClass(TextOutputFormat.class);

    hadoopJob.setMapperClass(MyHadoopMapper.class);
    hadoopJob.setReducerClass(MyHadoopReducer.class);

    hadoopJob.setCombinerClass(MyHadoopReducer.class);

    hadoopJob.setOutputKeyClass(Text.class);
    hadoopJob.setOutputValueClass(Text.class);

    System.exit(hadoopJob.waitForCompletion(true) ? 0: 1);
}

}

Answer 1

默认情况下，映射器输出类型为Text，而您使用floatwritable。这就是例外告诉你的。您需要指定映射器输出类型，如下所示：

job.setMapOutputValueClass(FloatWritable.class)