xixiNote

Appearance

Hive Install ​

TIP

hive安装需要先安装mysql数据库,未安装的请自行安装。

SoftWareVersiondownload
mysql5.7.24mysql5压缩包下载
hive2.3.4hive二进制包下载

其它版本的mysql和hive版本可自行到mysql官网和hive archive下载

前置 ​

  1. 关闭防火墙
sh
systemctl stop firewalld
systemctl disable firewalld
systemctl stop firewalld
systemctl disable firewalld

  1. 卸载linux系统自带MariaDB

    • 查询已安装的mariadb

      sh
      rpm -qa | grep mariadb
      rpm -qa | grep mariadb

    • 卸载查出的mariadb

      sh
      rpm -e --nodeps mariadb-libs-5.5.68-1.el7.x86_64
      rpm -e --nodeps mariadb-libs-5.5.68-1.el7.x86_64

解压hive安装包 ​

这里我将其放入了/opt/目录下，位置可以随意更改，这里以/opt/apache-hive-2.3.4-bin为例

sh
tar -xzvf apache-hive-2.3.4-bin.tar.gz -C /opt/
tar -xzvf apache-hive-2.3.4-bin.tar.gz -C /opt/

为其创建软链接:

sh
ln -s /opt/apache-hive-2.3.4-bin hive
ln -s /opt/apache-hive-2.3.4-bin hive

配置环境变量 ​

sh
vim /etc/profile
vim /etc/profile
sh
export HIVE_HOME=/opt/hive
export PATH=$HIVE_HOME/bin:$PATH
export HIVE_HOME=/opt/hive
export PATH=$HIVE_HOME/bin:$PATH

刷新配置文件使其生效:

sh
source /etc/profile
source /etc/profile

修改hive-site.xml配置文件 ​

将/opt/hive/conf文件夹下hive-default.xml.template文件复制一份，更名为hive-site.xml

sh
cp /opt/hive/conf/hive-default.xml.template /opt/hive/conf/hive-site.xml
cp /opt/hive/conf/hive-default.xml.template /opt/hive/conf/hive-site.xml

修改hive-site.xml文件，配置hive的元数据存储位置，这里使用的是mysql数据库.

初次进入会有很多内容，我们只需要找到并且配置其中的一些即可，如下：

  1. 配置mysql数据库连接:
xml
<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false</value>
  <description>
    JDBC connect string for a JDBC metastore.
    To use SSL to encrypt/authenticate the connection, provide database-specific SSL flagin the connection URL.
    For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
  </description>
</property>
<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false</value>
  <description>
    JDBC connect string for a JDBC metastore.
    To use SSL to encrypt/authenticate the connection, provide database-specific SSL flagin the connection URL.
    For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
  </description>
</property>
  1. 配置数据库驱动:
xml
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
  <description>Driver class name for a JDBC metastore</description>
</property>
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
  <description>Driver class name for a JDBC metastore</description>
</property>
  1. 配置数据库用户名为root:
xml
<property> 
    <name>javax.jdo.option.ConnectionUserName</name> 
    <value>root</value> 
    <description>Username to use against metastore database</description> 
</property>
<property> 
    <name>javax.jdo.option.ConnectionUserName</name> 
    <value>root</value> 
    <description>Username to use against metastore database</description> 
</property>
  1. 配置mysql数据库root密码:

这里的root密码需要和mysql数据库的root密码一致。

xml
<property> 
    <name>javax.jdo.option.ConnectionPassword</name> 
    <value>root</value> 
    <description>password to use against s database</description> 
</property>
<property> 
    <name>javax.jdo.option.ConnectionPassword</name> 
    <value>root</value> 
    <description>password to use against s database</description> 
</property>
  1. 验证元数据存储版本一致性，若默认为false，则不用修改:
xml
<property>
  <name>hive.metastore.schema.verification</name>
  <value>false</value>
  <description>
    Enforce metastore schema version consistency.
    True: Verify that version information stored in is compatible with one from Hivejars.  Also disable automatic
          schema migration attempt. Users are required to manually migrate schema afterHive upgrade which ensures
          proper metastore schema migration. (Default)
    False: Warn if the version information stored in metastore doesn't match with one fromin Hive jars.
  </description>
</property>
<property>
  <name>hive.metastore.schema.verification</name>
  <value>false</value>
  <description>
    Enforce metastore schema version consistency.
    True: Verify that version information stored in is compatible with one from Hivejars.  Also disable automatic
          schema migration attempt. Users are required to manually migrate schema afterHive upgrade which ensures
          proper metastore schema migration. (Default)
    False: Warn if the version information stored in metastore doesn't match with one fromin Hive jars.
  </description>
</property>
  1. 找到以下配置，修改对应的value值
xml
<property>
    <name>hive.querylog.location</name> 
    <value>/opt/hive/tmp</value> 
    <description>Location of Hive run time structured log file</description> 
</property>
<property>
    <name>hive.querylog.location</name> 
    <value>/opt/hive/tmp</value> 
    <description>Location of Hive run time structured log file</description> 
</property>
xml
<property>
    <name>hive.exec.local.scratchdir</name> 
    <value>/opt/hive/tmp</value>
</property>
<property>
    <name>hive.exec.local.scratchdir</name> 
    <value>/opt/hive/tmp</value>
</property>
xml
<property>
    <name>hive.downloaded.resources.dir</name> 
    <value>/opt/hive/tmp/resources</value>
</property>
<property>
    <name>hive.downloaded.resources.dir</name> 
    <value>/opt/hive/tmp/resources</value>
</property>
xml
<property>
    <name>hive.server2.logging.operation.log.location</name> 
    <value>/opt/hive/tmp/operation_logs</value> 
</property>
<property>
    <name>hive.server2.logging.operation.log.location</name> 
    <value>/opt/hive/tmp/operation_logs</value> 
</property>

在hive安装目录下创建临时文件夹tmp

sh
mkdir -p /opt/hive/tmp
mkdir -p /opt/hive/tmp

初始化hive元数据 ​

将mysql数据库驱动拷贝到hive安装目录下的lib目录下

这里的驱动jar包需自行到官网下载

sh
cp /opt/software/mysql-connector-java-5.1.47-bin.jar /opt/hive/lib/
cp /opt/software/mysql-connector-java-5.1.47-bin.jar /opt/hive/lib/

重启hadoop ​

sh
stop-all.sh
start-all.sh
stop-all.sh
start-all.sh

初始化数据库 ​

sh
schematool -initSchema -dbType mysql
schematool -initSchema -dbType mysql

出现如下提示则表示初始化成功

row
[root@master conf]# schematool -initSchema -dbType mysql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:  jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&useSSL=false
Metastore Connection Driver :  com.mysql.jdbc.Driver
Metastore connection User:  root
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Initialization script completed
schemaTool completed
[root@master conf]# schematool -initSchema -dbType mysql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:  jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&useSSL=false
Metastore Connection Driver :  com.mysql.jdbc.Driver
Metastore connection User:  root
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Initialization script completed
schemaTool completed

启动hive ​

sh
hive
hive

现在便可进入hive操作了

row
[root@master software]# hive
which: no hbase in (/opt/mysql/bin:/opt/hive/bin:/opt/jdk/bin:/opt/hadoop/bin:/opt/hadoop/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/apache-hive-2.3.4-bin/lib/hive-common-2.3.4.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>
[root@master software]# hive
which: no hbase in (/opt/mysql/bin:/opt/hive/bin:/opt/jdk/bin:/opt/hadoop/bin:/opt/hadoop/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/apache-hive-2.3.4-bin/lib/hive-common-2.3.4.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>