0%

ranger-spark插件部署

Posted on In

参考

1
2
https://github.com/yaooqinn/spark-authorizer
http://dl.bintray.com/spark-packages/maven/yaooqinn/spark-authorizer

部署Jar

将下面的包添加到$SPARK_HOME/jar中，切记千万要注意ranger-jars版本，最好是按照下图中的版本：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
eclipselink-2.5.2.jar
gson-2.2.4.jar
httpclient-4.5.1.jar
httpcore-4.4.4.jar
httpmime-4.5.2.jar
javax.persistence-2.1.0.jar
jersey-bundle-1.4.jar
jersey-client-2.22.2.jar
jersey-common-2.22.2.jar
jersey-container-servlet-2.22.2.jar
jersey-container-servlet-core-2.22.2.jar
jersey-guava-2.22.2.jar
jersey-media-jaxb-2.22.2.jar
jersey-server-2.22.2.jar
mybatis-3.2.8.jar
mysql-connector-java.jar
noggit-0.6.jar
ranger-hive-plugin-0.5.3.jar
ranger-plugins-audit-0.5.3.jar
ranger-plugins-common-0.5.3.jar
ranger-plugins-cred-0.5.3.jar

部署配置文件

  1. 将/etc/hive/2.6.5.0-292/0/conf.server中的ranger相关的配置文件拷贝到 SPARK_HOME/conf中,可以执行：find / -name xxx.xml 来找到；

    1
    2
    3
    4
    ranger-hive-audit.xml
    ranger-hive-security.xml
    ranger-policymgr-ssl.xml
    ranger-security.xml

  1. 并在ambari界面：spark配置中心：hive-site.xml中添加以下配置：

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    <property>
        <name>hive.security.authorization.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>hive.security.authorization.manager</name>
        <value>org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory</value>
    </property>

    <property>
        <name>hive.security.authenticator.manager</name>
        <value>org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator</value>
    </property>

    <property>
        <name>hive.conf.restricted.list</name>
        <value>hive.security.authorization.enabled,hive.security.authorization.manager,hive.security.authenticator.manager</value>
    </property>

  1. spark-default.xml添加以下配置：

    1
    spark.sql.extensions=org.apache.ranger.authorization.spark.authorizer.RangerSparkSQLExtension

spark job中的配置

需要在resource中添加：

1
2
3
hive-site.xml
ranger-hive-audit.xml
ranger-hive-security.xml

需要在程序中指定配置：

1
2
3
4
5
SparkConf sparkConf = new SparkConf();
sparkConf.set("spark.sql.extensions", "org.apache.ranger.authorization.spark.authorizer.RangerSparkSQLExtension");

//如果上面的resource中没有添加这些配置文件，则需要指定远程路径
sparkConf.set("spark.yarn.dist.files", "hdfs://xxx/ranger-hive-audit.xml,hdfs://xxx/ranger-hive-security.xml");