Avro模式演化

Question

我有两个问题：

>

我曾尝试使用模式V1编写记录，并使用模式V2读取记录，但出现以下错误：

org.apache.avro。AvroTypeException：找到foo，应为foo

我使用avro-1.7.3和：

   writer = new GenericDatumWriter<GenericData.Record>(SchemaV1);
   reader = new GenericDatumReader<GenericData.Record>(SchemaV2, SchemaV1);

以下是这两种模式的示例（我也尝试过添加命名空间，但没有成功）。

架构V1：

{
"name": "foo",
"type": "record",
"fields": [{
    "name": "products",
    "type": {
        "type": "array",
        "items": {
            "name": "product",
            "type": "record",
            "fields": [{
                "name": "a1",
                "type": "string"
            }, {
                "name": "a2",
                "type": {"type": "fixed", "name": "a3", "size": 1}
            }, {
                "name": "a4",
                "type": "int"
            }, {
                "name": "a5",
                "type": "int"
            }]
        }
    }
}]
}

架构V2：

{
"name": "foo",
"type": "record",
"fields": [{
    "name": "products",
    "type": {
        "type": "array",
        "items": {
            "name": "product",
            "type": "record",
            "fields": [{
                "name": "a1",
                "type": "string"
            }, {
                "name": "a2",
                "type": {"type": "fixed", "name": "a3", "size": 1}
            }, {
                "name": "a4",
                "type": "int"
            }, {
                "name": "a5",
                "type": "int"
            }]
        }
    }
},
{
            "name": "purchases",
            "type": ["null",{
                    "type": "array",
                    "items": {
                            "name": "purchase",
                            "type": "record",
                            "fields": [{
                                    "name": "a1",
                                    "type": "int"
                            }, {
                                    "name": "a2",
                                    "type": "int"
                            }]
                    }
            }]
}]
} 

提前谢谢。

Answer 1

最好的方法是使用模式映射来维护类似于模式的融合Avro模式注册表。

关键带走：

1.  Unlike Thrift, avro serialized objects do not hold any schema.
2.  As there is no schema stored in the serialized byte array, one has to provide the schema with which it was written.
3.  Confluent Schema Registry provides a service to maintain schema versions.
4.  Confluent provides Cached Schema Client, which checks in cache first before sending the request over the network.
5.  Json Schema present in “avsc” file is different from the schema present in Avro Object.
6.  All Avro objects extends from Generic Record
7.  During Serialization : based on schema of the Avro Object a schema Id is requested from the Confluent Schema Registry.
8.  The schemaId which is a INTEGER is converted to Bytes and prepend to serialized AvroObject.
9.  During Deserialization : First 4 bytes are removed from the ByteArray.  4 bytes are converted back to INTEGER(SchemaId)
10. Schema is requested from the Confluent Schema Registry and using this schema the byteArray is deserialized.

http://bytepadding.com/big-data/spark/avro/avro-serialization-de-serialization-using-confluent-schema-registry/

Answer 2

你可以做相反的事情。这意味着你可以解析数据模式1并从模式2写入数据。因为在写入时，它会将数据写入文件，如果我们在读取时不提供任何字段，那么它就可以了。但如果我们写入的字段少于读取的字段，那么它在读取时不会识别额外的字段，因此，它会给出错误。

Answer 3

我遇到了同样的问题。这可能是avro的错误，但您可能可以通过在“购买”字段中添加“默认”： null来解决问题。

查看我的博客了解详情：http://ben-tech.blogspot.com/2013/05/avro-schema-evolution.html