自定义WritableCompare将对象引用显示为输出

Question

我是Hadoop和Java的新手，我觉得我明显缺少了一些东西。如果这意味着什么的话，我正在使用Hadoop 1.0.3。

我使用hadoop的目标是获取一堆文件并一次解析一个文件（而不是逐行）。每个文件将生成多个键值，但其他行的上下文很重要。键和值是多值/复合的，因此我为键实现了可写比较，为值实现了可写。由于每个文件的处理需要一点 CPU，因此我想保存映射器的输出，然后稍后运行多个化简器。

对于复合键，我遵循[http://stackoverflow.com/questions/12427090/hadoop-composite-key][1]

问题是，输出只是Java对象引用，而不是组合键和值。示例:< code > linkkey writable @ bd2f 9730 LinkValueWritable @ 8752408 c

我不确定这个问题是与根本没有减少数据有关还是

这是我的主要课程:

public static void main(String[] args) throws Exception {
  JobConf conf = new JobConf(Parser.class);
  conf.setJobName("raw_parser");

  conf.setOutputKeyClass(LinkKeyWritable.class);
  conf.setOutputValueClass(LinkValueWritable.class);

  conf.setMapperClass(RawMap.class);
  conf.setNumMapTasks(0);

  conf.setInputFormat(PerFileInputFormat.class);
  conf.setOutputFormat(TextOutputFormat.class);

  PerFileInputFormat.setInputPaths(conf, new Path(args[0]));
  FileOutputFormat.setOutputPath(conf, new Path(args[1]));

  JobClient.runJob(conf);
}

还有我的Mapper类：

公共类RawMap扩展MapReduceBase实现映射器{

    public void map(NullWritable key, Text value,
            OutputCollector<LinkKeyWritable, LinkValueWritable> output,
            Reporter reporter) throws IOException {
        String json = value.toString();
        SerpyReader reader = new SerpyReader(json);
        GoogleParser parser = new GoogleParser(reader);
        for (String page : reader.getPages()) {
            String content = reader.readPageContent(page);
            parser.addPage(content);
        }
        for (Link link : parser.getLinks()) {
            LinkKeyWritable linkKey = new LinkKeyWritable(link);
            LinkValueWritable linkValue = new LinkValueWritable(link);
            output.collect(linkKey, linkValue);
        }
    }
}

链接基本上是各种信息的结构，在LinkKeyWritable和LinkValueWritable之间拆分

LinkKeyWritable:

public class LinkKeyWritable implements WritableComparable<LinkKeyWritable>{
    protected Link link;

    public LinkKeyWritable() {
        super();
        link = new Link();
    }

    public LinkKeyWritable(Link link) {
        super();
        this.link = link;
    }

    @Override
    public void readFields(DataInput in) throws IOException {
        link.batchDay = in.readLong();
        link.source = in.readUTF();
        link.domain = in.readUTF();
        link.path = in.readUTF();
    }

    @Override
    public void write(DataOutput out) throws IOException {
        out.writeLong(link.batchDay);
        out.writeUTF(link.source);
        out.writeUTF(link.domain);
        out.writeUTF(link.path);
    }

    @Override
    public int compareTo(LinkKeyWritable o) {
        return ComparisonChain.start().
                compare(link.batchDay, o.link.batchDay).
                compare(link.domain, o.link.domain).
                compare(link.path, o.link.path).
                result();
    }

    @Override
    public int hashCode() {
        return Objects.hashCode(link.batchDay, link.source, link.domain, link.path);
    }

    @Override
    public boolean equals(final Object obj){
        if(obj instanceof LinkKeyWritable) {
            final LinkKeyWritable o = (LinkKeyWritable)obj;
            return Objects.equal(link.batchDay, o.link.batchDay)
                    && Objects.equal(link.source, o.link.source)
                    && Objects.equal(link.domain, o.link.domain)
                    && Objects.equal(link.path, o.link.path);
        }
        return false;
    }
}

LinkValueWritable：

public class LinkValueWritable implements Writable{
    protected Link link;

    public LinkValueWritable() {
        link = new Link();
    }

    public LinkValueWritable(Link link) {
        this.link = new Link();
        this.link.type = link.type;
        this.link.description = link.description;
    }

    @Override
    public void readFields(DataInput in) throws IOException {
        link.type = in.readUTF();
        link.description = in.readUTF();
    }

    @Override
    public void write(DataOutput out) throws IOException {
        out.writeUTF(link.type);
        out.writeUTF(link.description);
    }

    @Override
    public int hashCode() {
        return Objects.hashCode(link.type, link.description);
    }

    @Override
    public boolean equals(final Object obj){
        if(obj instanceof LinkKeyWritable) {
            final LinkKeyWritable o = (LinkKeyWritable)obj;
            return Objects.equal(link.type, o.link.type)
                    && Objects.equal(link.description, o.link.description);
        }
        return false;
    }
}

Answer 1

看起来你有一个你想要的对象列表。如果你想打印出一个人类可读的版本而不是一个丑陋的java引用，你需要在你的可写上实现toString（）。

Answer 2

我认为答案在于TextOutputFormat的实现。具体来说，LineRecordWriter的writeObject方法：

/**
 * Write the object to the byte stream, handling Text as a special
 * case.
 * @param o the object to print
 * @throws IOException if the write throws, we pass it on
 */
private void writeObject(Object o) throws IOException {
  if (o instanceof Text) {
    Text to = (Text) o;
    out.write(to.getBytes(), 0, to.getLength());
  } else {
    out.write(o.toString().getBytes(utf8));
  }
}

如您所见，如果您的键或值不是Text对象，它会调用toString方法并将其写出。由于您在键和值中未实现toString，因此它使用Object类的实现，即写出引用。

我认为您应该尝试编写一个合适的toString函数或使用不同的OutputFormat。