Kafka Connect JDBC：Oracle数字字段的精度为64，无法从Spark中读取

Question

{
  "type": "record",
  "name": "ConnectDefault",
  "namespace": "io.confluent.connect.avro",
  "fields": [
    {
      "name": "ID",
      "type": [
        "null",
        {
          "type": "bytes",
          "scale": 0,
          "precision": 64,
          "connect.version": 1,
          "connect.parameters": {
            "scale": "0"
          },
          "connect.name": "org.apache.kafka.connect.data.Decimal",
          "logicalType": "decimal"
        }
      ],
      "default": null
    }....
}

原因：org.apache.spark.sql.analysisException：decimal只能支持高达38的精度；在org.apache.spark.sql.types.decimaltype.(decimaltype.scala:51)在org.apache.spark.sql.avro.schemaConverters$.TosqlTypeHelper(SchemaConverters.scala:60)在org.apache.spark.sql.avro.schemaConverters$.TosqlTypeHelper(SchemaConverters.scala:105)在org.apache.spark.sql.avro.schemaConverters Scala:81)在Scala.collection.traversablelike$$anonfun$map$1.在Scala.collection.traversablelike.scala:234)在Scala.collection.traversablelike$$anonfun$map$1.在Scala.collection.traversablelike.scala:234)在Scala.collection.iterator$class.foreach(iterator.scala:891)在Scala.collection.abstraction.foreach(iterator.scala:1334)在.map(traversablelike.scala:234)在scala.collection.abstractTraversable.map(traversable.scala:104)在在org.apache.spark.sql.avro.schemaConverters$.TosqlTypeHelper(SchemaConverters.scala:81)在org.apache.spark.sql.avro.schemaConverters$.TosqlType(SchemaConverters.scala:46)在org.apache.spark.sql.avro.avrofileFormat.InferSchema(avrofileFormat.scala:93)在org.apache.spark.sql.execution.datasource$$AnonFun$6 6.apply(datasource.scala:180)

我读取AVO文件的代码片段是：

def readSchemaOfAvroPartition(avroLocation: String, partitionColumn: String, partitionValue: String): StructType = {
      sparkSession.read.format(AVRO)
        .load(s"${avroLocation}/${partitionColumn}=${partitionValue}")
        .schema
    }

根据Oracle文档的最大精度应该是38。https://docs.oracle.com/cd/b28359_01/server.111/b28318/datatype.htm#cncpt313

如何强制Kafka Connect将此模式值注册为38而不是64？

Answer 1

这不一定是Kafka连接器中的bug，但总的来说Kafka是如何工作的。在许多数据库中，数值或十进制列是用精度和小数位数值定义的。基于数据库技术，数据库缺省了精度和规模。

Kafka connect没有很好的方法来处理跨数据库和其他非数据库文件系统的问题。

更多细节可以在这里找到--