Apache CarbonData Dev Mailing List archive

Got kerberos login error while starting CarbonData Thrift Server

classic Classic list List threaded Threaded
♦
♦
 2 messages Options
冉明轩
Reply | Threaded
Open this post in threaded view
♦
♦
|

Got kerberos login error while starting CarbonData Thrift Server

冉明轩
Hello all,


I am new to carbondata and just about to start a CarbonData Thrift Server to have a try on its through put.


Environment:
Cluster: Cloudera CDH5.5.3 with kerberos installed as principle + keytab style
Spark: 2.2.1
Carbondata: apache-carbondata-1.4.0-bin-spark2.2.1-hadoop2.7.2.jar


Process:
1. build carbondata by following https://github.com/apache/carbondata/blob/master/build/README.md
2. Install and configure carbondata on Spark on Yarn Cluster by following http://carbondata.apache.org/installation-guide.html
3. create file /home/mingxuan/spark-2.2.1-bin-hadoop2.7/bin/carbon-thrift.sh:
    #!/bin/bash
    /home/mingxuan/spark-2.2.1-bin-hadoop2.7/bin/spark-submit \
        --class org.apache.carbondata.spark.thriftserver.CarbonThriftServer \
        --principal mingxuan/[hidden email] \
        --keytab /home/mingxuan/mingxuan.keytab \
        --queue root.mingxuan \
        --master yarn \
        --deploy-mode client \
        --driver-memory 8g \
        --executor-cores 4 \
        --executor-memory 10g \
        --num-executors 10 \
    /home/mingxuan/spark-2.2.1-bin-hadoop2.7/carbonlib/apache-carbondata-1.4.0-bin-spark2.2.1-hadoop2.7.2.jar
    hdfs://nsha/user/hive/warehouse/ranmx.db/carbondata
4. cd to spark dir and run command: sh /home/mingxuan/spark-2.2.1-bin-hadoop2.7/bin/carbon-thrift.sh


Error Log:
18/07/10 16:08:33 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is /user/hive/warehouse
Exception in thread "main" org.apache.hive.service.ServiceException: Unable to login to kerberos with given principal/keytab
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIService.init(SparkSQLCLIService.scala:58)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$$anonfun$initCompositeService$1.apply(SparkSQLCLIService.scala:79)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$$anonfun$initCompositeService$1.apply(SparkSQLCLIService.scala:79)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$class.initCompositeService(SparkSQLCLIService.scala:79)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.initCompositeService(HiveThriftServer2.scala:272)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.init(HiveThriftServer2.scala:292)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.startWithContext(HiveThriftServer2.scala:64)
at org.apache.carbondata.spark.thriftserver.CarbonThriftServer$.main(CarbonThriftServer.scala:93)
at org.apache.carbondata.spark.thriftserver.CarbonThriftServer.main(CarbonThriftServer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: HiveServer2 Kerberos principal or keytab is not correctly configured
at org.apache.hive.service.auth.HiveAuthFactory.loginFromKeytab(HiveAuthFactory.java:197)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIService.init(SparkSQLCLIService.scala:53)
... 21 more


Does anybody has any idea about this exception? Does hadoop 2.6 matters?


Ran Mingxuan
ran gabriele
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Got kerberos login error while starting CarbonData Thrift Server

ran gabriele
This is due to the fact that starting a hive server 2 is using kerberos principle different from what you used to submit the spark task.
This happens when the hive.xml file have some specification on kerberos.
In this case, make sure that keytab address and principal are both described in hive.xml file.


Sent from Outlook<http://aka.ms/weboutlook>

________________________________
From: 冉明轩 <[hidden email]>
Sent: Wednesday, July 11, 2018 5:31 AM
To: [hidden email]; [hidden email]
Subject: Got kerberos login error while starting CarbonData Thrift Server

Hello all,


I am new to carbondata and just about to start a CarbonData Thrift Server to have a try on its through put.


Environment:
Cluster: Cloudera CDH5.5.3 with kerberos installed as principle + keytab style
Spark: 2.2.1
Carbondata: apache-carbondata-1.4.0-bin-spark2.2.1-hadoop2.7.2.jar


Process:
1. build carbondata by following https://github.com/apache/carbondata/blob/master/build/README.md
2. Install and configure carbondata on Spark on Yarn Cluster by following http://carbondata.apache.org/installation-guide.html
3. create file /home/mingxuan/spark-2.2.1-bin-hadoop2.7/bin/carbon-thrift.sh:
    #!/bin/bash
    /home/mingxuan/spark-2.2.1-bin-hadoop2.7/bin/spark-submit \
        --class org.apache.carbondata.spark.thriftserver.CarbonThriftServer \
        --principal mingxuan/[hidden email] \
        --keytab /home/mingxuan/mingxuan.keytab \
        --queue root.mingxuan \
        --master yarn \
        --deploy-mode client \
        --driver-memory 8g \
        --executor-cores 4 \
        --executor-memory 10g \
        --num-executors 10 \
    /home/mingxuan/spark-2.2.1-bin-hadoop2.7/carbonlib/apache-carbondata-1.4.0-bin-spark2.2.1-hadoop2.7.2.jar
    hdfs://nsha/user/hive/warehouse/ranmx.db/carbondata
4. cd to spark dir and run command: sh /home/mingxuan/spark-2.2.1-bin-hadoop2.7/bin/carbon-thrift.sh


Error Log:
18/07/10 16:08:33 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is /user/hive/warehouse
Exception in thread "main" org.apache.hive.service.ServiceException: Unable to login to kerberos with given principal/keytab
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIService.init(SparkSQLCLIService.scala:58)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$$anonfun$initCompositeService$1.apply(SparkSQLCLIService.scala:79)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$$anonfun$initCompositeService$1.apply(SparkSQLCLIService.scala:79)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$class.initCompositeService(SparkSQLCLIService.scala:79)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.initCompositeService(HiveThriftServer2.scala:272)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.init(HiveThriftServer2.scala:292)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.startWithContext(HiveThriftServer2.scala:64)
at org.apache.carbondata.spark.thriftserver.CarbonThriftServer$.main(CarbonThriftServer.scala:93)
at org.apache.carbondata.spark.thriftserver.CarbonThriftServer.main(CarbonThriftServer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: HiveServer2 Kerberos principal or keytab is not correctly configured
at org.apache.hive.service.auth.HiveAuthFactory.loginFromKeytab(HiveAuthFactory.java:197)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIService.init(SparkSQLCLIService.scala:53)
... 21 more


Does anybody has any idea about this exception? Does hadoop 2.6 matters?


Ran Mingxuan
Free forum by Nabble Edit this page