druid.io mysql_Druid：Druid.io 部署&使用文档

参考：

1.集群模式下部署

Prerequisites : Java 7 or higher & Zookeeper & mysql

下载Druid.io :

curl -O http://static.druid.io/artifacts/releases/druid-0.9.1.1-bin.tar.gz

tar -xzf druid-0.9.1.1-bin.tar.gz

cd druid-0.9.1.1

文件夹目录结构 ：

LICENSE - the license files.

bin/ - scripts related to the single-machine quickstart.

conf/* - template configurations for a clustered setup.

conf-quickstart/* - configurations for the single-machine quickstart.

extensions/* - all Druid extensions.

hadoop-dependencies/* - Druid Hadoop dependencies.

lib/* - all included software packages for core Druid.

quickstart/* - files related to the single-machine quickstart.

所有配置文件均在 conf/* 目录下.

配饰HDFS为Druid.io的deep storage & 配置zk & 配置mysql

修改 conf/druid/_common/common.runtime.properties 文件.

#

# Extensions

#

# This is not the full list of Druid extensions, but common ones that people often use. You may need to change this list

# based on your particular setup.

#使用 "mysql-metadata-storage" 作为metadata的存储

#使用 "druid-hdfs-storage" 作为 deep storage

#使用 "druid-parquet-extensions" 向druid中插入parquet数据

druid.extensions.loadList=["druid-kafka-eight", "druid-histogram", "druid-datasketches", "mysql-metadata-storage", "druid-hdfs-storage", "druid-avro-extensions", "druid-parquet-extensions"]

# If you have a different version of Hadoop, place your Hadoop client jar files in your hadoop-dependencies directory

# and uncomment the line below to point to your directory.

#druid.extensions.hadoopDependenciesDir=/my/dir/hadoop-dependencies

#

# Logging

#

# Log all runtime properties on startup. Disable to avoid logging properties on startup:

druid.startup.logging.logProperties=true

#

# Zookeeper

#

druid.zk.service.host=tagtic-slave01:2181,tagtic-slave02:2181,tagtic-slave03:2181

druid.zk.paths.base=/druid

#

# Metadata storage

#

# For MySQL:

druid.metadata.storage.type=mysql

druid.metadata.storage.connector.connectURI=jdbc:mysql://tagtic-master:3306/druid

druid.metadata.storage.connector.user=root

druid.metadata.storage.connector.password=123456

#

# Deep storage

#

# For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp):

druid.storage.type=hdfs

druid.storage.storageDirectory=/druid/segments

#

# Indexing service logs

#

# For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp):

druid.indexer.logs.type=hdfs

druid.indexer.logs.directory=/druid/indexing-logs

#

# Service discovery

#

druid.selectors.indexing.serviceName=druid/overlord

druid.selectors.coordinator.serviceName=druid/coordinator

#

# Monitoring

#

druid.monitoring.monitors=["com.metamx.metrics.JvmMonitor"]

druid.emitter=logging

druid.emitter.logging.logLevel=info

将 Hadoop 的配置文件(core-site.xml, hdfs-site.xml, yarn-site.xml, mapred-site.xml) cp 到 conf/druid/_common 目录下

修改 conf/druid/middleManager/runtime.properties 文件.

druid.service=druid/middleManager

druid.port=18091

# Number of tasks per middleManager

druid.worker.capacity=3

# Task launch parameters

# **CDH版本添加 -Dhadoop.mapreduce.job.classloader=true 来解决hadoop indexer导入时jar包冲突问题**

druid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dhadoop.mapreduce.job.classloader=true

druid.indexer.task.baseTaskDir=var/druid/task

# HTTP server threads

druid.server.http.numThreads=25

# Processing threads and buffers

druid.processing.buffer.sizeBytes=536870912

druid.processing.numThreads=2

# Hadoop indexing

druid.indexer.task.hadoopWorkingPath=/tmp/druid-indexing

druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.6.0"]

Hadoop集群版本必须和Druid.io中版本同一，可以通过pull-deps下载相同hadoop-dependencies版本,e.g. :

java -classpath "lib/*" io.druid.cli.Main tools pull-deps --defaultVersion 0.9.1.1 -c io.druid.extensions:mysql-metadata-storage:0.9.1.1 -c druid-hdfs-storage -h org.apache.hadoop:hadoop-client:2.6.0

项目中Druid.io配置端口号：

druid.service=druid/coordinator druid.port=18081

druid.service=druid/broker druid.port=18082

druid.service=druid/historical druid.port=18083

druid.service=druid/overlord druid.port=18090

druid.service=druid/middleManager druid.port=18091

2.启动Druid.io

java `cat conf/druid/coordinator/jvm.config | xargs` -cp conf/druid/_common:conf/druid/coordinator:lib/* io.druid.cli.Main server coordinator &>> logs/coordinator.log &

java `cat conf/druid/overlord/jvm.config | xargs` -cp conf/druid/_common:conf/druid/overlord:lib/* io.druid.cli.Main server overlord &>> logs/overlord.log &

java `cat conf/druid/historical/jvm.config | xargs` -cp conf/druid/_common:conf/druid/historical:lib/* io.druid.cli.Main server historical &>> logs/historical.log &

java `cat conf/druid/middleManager/jvm.config | xargs` -cp conf/druid/_common:conf/druid/middleManager:lib/* io.druid.cli.Main server middleManager &>> logs/middleManager.log &

java `cat conf/druid/broker/jvm.config | xargs` -cp conf/druid/_common:conf/druid/broker:lib/* io.druid.cli.Main server broker &>> logs/broker.log &

3.从HDFS导入数据到Druid.io

解决传入数据时区问题 Hadoop Configuration，在conf/druid/_common/mapred-site.xml中添加

mapreduce.map.java.opts

-server -Xmx1536m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

mapreduce.reduce.java.opts

-server -Xmx2560m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps