hadoop中用相同键减去两个数

Question

key2值2

...

文件2：

我让映射写键，值用一个字符预先说明它来自file1或file2，但不确定如何编写reduce阶段

我的map方法是

public void map(LongWritable key,Text val,Context context) throws IOException,     InterruptedException
    {
        Text outputKey = new Text();
        Text outputValue = new Text();
        outputKey.set(key.toString());
        if ("A")
        {               
            outputValue.set("A,"+val);
        }
        else
        {
            outputValue.set("B," + val);
        }
        context.write(outputKey,  outputValue);
    }
}

Answer 1

它应该足够简单，因为您已经标记了它，尽管开始时有点混乱。我假设发出的值类似于A23(file1)&B139(file2)。片段：

public void reduce(Text key, Iterable<Text> values, Context context)
        throws IOException, InterruptedException {

    int diff = 0;
    int denominator = 1;
    for (Text val : values) {
        if (val.toString().startsWith("A")) {
            denominator = Integer.parseInt(val.toString().substring(1));
            diff += denominator;
        } else if (val.toString().startsWith("B")) {
            diff -= Integer.parseInt(val.toString().substring(1));
        } else {
            // This block shouldn't be reached unless malformed values are emitted
            // Throw an exception or log it
        }
    }
    diff /= denominator;
    context.write(key, new IntWritable(diff));
}

希望这能有所帮助。但我认为，当key1和key2相等时，您的方法将严重失败。

更新
映射应如下所示，以便使用上述简化程序：

public void map(LongWritable key, Text val, Context context)
            throws IOException, InterruptedException {
        String fileName = ((FileSplit) context.getInputSplit()).getPath().getName();
        String[] keyVal = val.toString().split("\\s+");
        Text outputKey = new Text(keyVal[0]);
        Text outputValue = new Text();
        outputKey.set(key.toString());
        if ("fileA".equals(fileName)) {
            outputValue.set("A" + keyVal[1]);
        } else {
            outputValue.set("B" + keyVal[1]);
        }
        context.write(outputKey, outputValue);
    }