火花作业中的Kryo序列化错误

Question

public class SerializeTest {

    public static class Toto implements Serializable {
        private static final long serialVersionUID = 6369241181075151871L;
        private String a;

        public String getA() {
            return a;
        }

        public void setA(String a) {
            this.a = a;
        }
    }

    private static final PairFunction<Toto, Toto, Integer> WRITABLE_CONVERTOR = new PairFunction<Toto, Toto, Integer>() {
        private static final long serialVersionUID = -7119334882912691587L;

        @Override
        public Tuple2<Toto, Integer> call(Toto input) throws Exception {
            return new Tuple2<Toto, Integer>(input, 1);
        }
    };

    public static void main(String[] args) {
        SparkConf conf = new SparkConf().setAppName("SerializeTest");
        conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
        conf.registerKryoClasses(new Class<?>[]{Toto[].class});
        JavaSparkContext context = new JavaSparkContext(conf);

        List<Toto> list = new ArrayList<Toto>();
        list.add(new Toto());
        JavaRDD<Toto> cursor = context.parallelize(list, list.size());

        JavaPairRDD<Toto, Integer> writable = cursor.mapToPair(WRITABLE_CONVERTOR);
        writable.saveAsHadoopFile(args[0], Toto.class, Integer.class, SequenceFileOutputFormat.class);

        context.close();
    }

}

IOException：找不到键类'com.test.serializetest.toto'的序列化程序。如果使用自定义序列化，请确保配置“io.serializations”配置正确。在org.apache.hadoop.io.sequenceFile$writer.init(sequenceFile.java:1179)在org.apache.hadoop.io.sequenceFile$writer.（sequenceFile.java:1094)在org.apache.hadoop.io.sequenceFile.createWriter(sequenceFile.java:1094)在org.apache.hadoop.io.sequenceFile.createWriter(sequenceFile.java:273)在在org.apache.spark.rdd.pairrddfunctions$$anonfun$13.在org.apache.spark.rdd.pairrddfunctions.scala:1068)在org.apache.spark.rdd.pairrddfunctions$$anonfun$13.在org.apache.spark.schedure.resulttask.runtask(resulttask.scala:1059)在org.apache.spark.schedure.task.run(task.scala:61)在203）在java.util.concurrent.ThreadPoolExecutor.runworker(ThreadPoolExecutor.java:1142)在java.util.concurrent.ThreadPoolExecutor$worker.run(ThreadPoolExecutor.java:617)在java.lang.thread.run（thread.java：745）15/09/21 17:49:14 WARN TaskSetManager：在stage 0.0（TID 0,localhost）中丢失了任务0.0：java.io.ioException：找不到键类的序列化程序：“com.test.serializetest.toto”。如果使用自定义序列化，请确保配置“io.serializations”配置正确。在org.apache.hadoop.io.sequenceFile$writer.init(sequenceFile.java:1179)在org.apache.hadoop.io.sequenceFile$writer.（sequenceFile.java:1094)在org.apache.hadoop.io.sequenceFile.createWriter(sequenceFile.java:1094)在org.apache.hadoop.io.sequenceFile.createWriter(sequenceFile.java:273)在在org.apache.spark.rdd.pairrddfunctions$$anonfun$13.在org.apache.spark.rdd.pairrddfunctions.scala:1068)在org.apache.spark.rdd.pairrddfunctions$$anonfun$13.在org.apache.spark.schedure.resulttask.runtask(resulttask.scala:1059)在org.apache.spark.schedure.task.run(task.scala:61)在203）在java.util.concurrent.ThreadPoolExecutor.runworker(ThreadPoolExecutor.java:1142)在java.util.concurrent.ThreadPoolExecutor$worker.run(ThreadPoolExecutor.java:617)在java.lang.thread.run处(thread.java:745)

谢了。

Answer 1

这个错误既与Spark无关，也与Kryo无关。

使用Hadoop输出格式时，需要确保键和值是可写的实例。Hadoop默认情况下不使用Java序列化（您也不想使用它，因为它非常无效）

您可以在配置中检查io.serializations属性，您将看到使用的序列化程序列表，包括org.apache.hadoop.io.serializer.writableSerialization