Apache Druid 解析ORC及parquet格式的数据

Apache Druid可以从本地或者HDFS批量摄取数据，现在最新版本（0.18）也支持直接解析ORC及parquet格式的数据，但是要使用这个功能还需要进行简单的配置。

官方文档说明

Apache Druid打包了所有的核心扩展（参考本文附件），您可以通过将需要的扩展名添加到common.runtime.properties中的druid.extensions.loadList。例如，要加载postqresql-metadata-storage和druid-hdfs-storage扩展，请使用配置：

druid.extensions.loadList=["postgresql-metadata-storage", "druid-hdfs-storage"]

所以当我们需要Druid 解析ORC及Parquet格式的数据时，就需要这样配置：

druid.extensions.loadList=["druid-hdfs-storage", "druid-kafka-indexing-service", "druid-datasketches","druid-orc-extensions","druid-parquet-extensions"]

配置好后，重启集群即可使用。

---

附件

Apache Druid内置的核心扩展主要有：

Name	Description	Docs
druid-avro-extensions	Support for data in Apache Avro data format.	link
druid-azure-extensions	Microsoft Azure deep storage.	link
druid-basic-security	Support for Basic HTTP authentication and role-based access control.	link
druid-bloom-filter	Support for providing Bloom filters in druid queries.	link
druid-datasketches	Support for approximate counts and set operations with Apache DataSketches.	link
druid-google-extensions	Google Cloud Storage deep storage.	link
druid-hdfs-storage	HDFS deep storage.	link
druid-histogram	Approximate histograms and quantiles aggregator. Deprecated, please use the DataSketches quantiles aggregator from the druid-datasketches extension instead.	link
druid-kafka-extraction-namespace	Apache Kafka-based namespaced lookup. Requires namespace lookup extension.	link
druid-kafka-indexing-service	Supervised exactly-once Apache Kafka ingestion for the indexing service.	link
druid-kinesis-indexing-service	Supervised exactly-once Kinesis ingestion for the indexing service.	link
druid-kerberos	Kerberos authentication for druid processes.	link
druid-lookups-cached-global	A module for lookups providing a jvm-global eager caching for lookups. It provides JDBC and URI implementations for fetching lookup data.	link
druid-lookups-cached-single	Per lookup caching module to support the use cases where a lookup need to be isolated from the global pool of lookups	link
druid-orc-extensions	Support for data in Apache ORC data format.	link
druid-parquet-extensions	Support for data in Apache Parquet data format. Requires druid-avro-extensions to be loaded.	link
druid-protobuf-extensions	Support for data in Protobuf data format.	link
druid-ranger-security	Support for access control through Apache Ranger.	link
druid-s3-extensions	Interfacing with data in AWS S3, and using S3 as deep storage.	link
druid-ec2-extensions	Interfacing with AWS EC2 for autoscaling middle managers	UNDOCUMENTED
druid-stats	Statistics related module including variance and standard deviation.	link
mysql-metadata-storage	MySQL metadata store.	link
postgresql-metadata-storage	PostgreSQL metadata store.	link
simple-client-sslcontext	Simple SSLContext provider module to be used by Druid's internal HttpClient when talking to other Druid processes over HTTPS.	link
druid-pac4j	OpenID Connect authentication for druid processes.	link

另外还有第三方社区提供的扩展：

Name	Description	Docs
aliyun-oss-extensions	Aliyun OSS deep storage	link
ambari-metrics-emitter	Ambari Metrics Emitter	link
druid-cassandra-storage	Apache Cassandra deep storage.	link
druid-cloudfiles-extensions	Rackspace Cloudfiles deep storage and firehose.	link
druid-distinctcount	DistinctCount aggregator	link
druid-redis-cache	A cache implementation for Druid based on Redis.	link
druid-time-min-max	Min/Max aggregator for timestamp.	link
sqlserver-metadata-storage	Microsoft SQLServer deep storage.	link
graphite-emitter	Graphite metrics emitter	link
statsd-emitter	StatsD metrics emitter	link
kafka-emitter	Kafka metrics emitter	link
druid-thrift-extensions	Support thrift ingestion	link
druid-opentsdb-emitter	OpenTSDB metrics emitter	link
materialized-view-selection, materialized-view-maintenance	Materialized View	link
druid-moving-average-query	Support for Moving Average and other Aggregate Window Functions in Druid queries.	link
druid-influxdb-emitter	InfluxDB metrics emitter	link
druid-momentsketch	Support for approximate quantile queries using the momentsketch library	link
druid-tdigestsketch	Support for approximate sketch aggregators based on T-Digest	link
gce-extensions	GCE Extensions	link