apache-hive-3.1.2安装和部署

商迪

2023-12-01

hive的安装：

1、下载apache-hive-3.1.2-bin.tar.gz

2、解压并配置环境变量

a、解压包并重命名为hive-3.1.2

     [root@hadoop-01 local]# tar -zxvf apache-hive-3.1.2-bin.tar.gz 
     [root@hadoop-01 local]# mv apache-hive-3.1.2-bin hive-3.1.2

b、配置环境变量vim /etc/profile

添加HIVE_HOME目录

export HIVE_HOME=/usr/local/hive-3.1.2

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin

[root@hadoop-01 local]# vim /etc/profile
[root@hadoop-01 local]# source /etc/profile

3、修改hive配置

复制hive-3.1.2/conf目录下hive-default.xml.template 为hive-site.xml并修改

a、修改数据库配置，这里使用的是mysql

添加驱动包到hive-3.1.2的lib目录下；我使用的是mysql-connector-java-5.1.30.jar

  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>123456</value>
    <description>数据库密码</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://192.168.15.50:3306/hive_metadata?createDatabaseIfNotExist=true</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>
<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
    <description>Username to use against metastore database</description>
  </property>

注意：创建数据源时要求字符集使用latin1；否则报异常“MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes” 创建hive的mysql数据库，如：“create database hive_metadata character set latin1;”

b、配置hive自动创建表，改模式可两种模式

第一种：直接修改datanucleus.schema.autoCreateAll 为true；其默认值为false


 <property>
    <name>datanucleus.schema.autoCreateAll</name>
    <value>true</value>
    <description>Auto creates necessary schema on a startup if one doesn't exist. Set this to false, after creating it once.To enable auto create also set hive.metastore.schema.verification=false. Auto creation is not recommended for production use cases, run schematool command instead.</description>
  </property>

第二种：添加表的自动创建属性

  <property>
         <name>datanucleus.schema.autoCreateSchema</name>
         <value>true</value>
    </property>
    <property>
         <name>datanucleus.schema.autoCreateSchema</name>
         <value>true</value>
    </property>

    <property>
        <name>datanucleus.schema.autoCreateTables</name>
        <value>true</value>
    </property>

    <property>
        <name>datanucleus.schema.autoCreateColumns</name>
        <value>true</value>
    </property>
   <property>
   <name>datanucleus.fixedDatastore</name>
   <value>false</value>
</property>

3、修改${system:java.io.tmpdir}的路径

hive.exec.local.scratchdir 修改成/tmp/hive;hive.downloaded.resources.修改成/tmp/hive/resources;hive.querylog.location修改成./tmp/hive/query/log

hive.server2.logging.operation.log.location修改成/tmp/hive/operation_logs

  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>${system:java.io.tmpdir}/${system:user.name}</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
  <property>
    <name>hive.downloaded.resources.dir</name>
    <value>${system:java.io.tmpdir}/${hive.session.id}_resources</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
  </property>
  <property>
    <name>hive.querylog.location</name>
    <value>${system:java.io.tmpdir}/${system:user.name}</value>
    <description>Location of Hive run time structured log file</description>
  </property>
 <property>
    <name>hive.server2.logging.operation.log.location</name>
    <value>${system:java.io.tmpdir}/${system:user.name}/operation_logs</value>
    <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
  </property>

经过修改后可得到

  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/tmp/hive</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
  <property>
    <name>hive.downloaded.resources.dir</name>
    <value>/tmp/hive/resources</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
  </property>
  <property>
    <name>hive.querylog.location</name>
    <value>/tmp/hive/query/log</value>
    <description>Location of Hive run time structured log file</description>
  </property>
 <property>
    <name>hive.server2.logging.operation.log.location</name>
    <value>/tmp/hive/operation_logs</value>
    <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
  </property>

4、最后启动bin目录下hive

注意：如果show tables；异常“FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient”时，执行schematool -dbType mysql -initSchema；如

[root@hadoop02 bin]# ./hive
which: no hbase in (/usr/local/java/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/zookeeper-3.4.14/bin:/usr/local/hadoop/bin:/usr/local/hadoop/sbin:/usr/local/flume/bin:/usr/local/kafka/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 6e0e8815-5256-4995-8af3-a7a5818f43f8

Logging initialized using configuration in jar:file:/usr/local/hive-3.1.2/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> show tables;
FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
hive> show databases;
FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
hive> exit
    > ;
[root@hadoop02 bin]# ls
beeline  ext  hive  hive-config.sh  hiveserver2  hplsql  init-hive-dfs.sh  metatool  schematool
[root@hadoop02 bin]# ./schematool -dbType mysql -initSchema

apache-hive-3.1.2安装和部署

相关阅读

相关文章

相关问答

相关文档