|
|
Dear all,
I use HDP 3.1.4 (Hadoop 3.1.1, Hive 3.1.0, Spark 2.3.2).
I compiled CarbonData 2.0 (Current, downloaded at 04.03.2020) with
mvn clean -DskipTests -Pbuild-with-format -Pspark-2.3 -Phadoop-2.8 package
and distributed it around cluster (following the documentation).
Next, I started spark-shell with
spark-shell --master yarn --driver-memory 1G --executor-memory 2G --executor-cores 2
After that, I created CarbonData object (without any errors):
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.CarbonSession._
val rootPath = "/carbon/data"
val storeLocation = s"$rootPath/store"
val warehouse = s"$rootPath/warehouse"
val metaStoreDB = s"$rootPath/metastore_db"
val carbon = SparkSession.builder().enableHiveSupport().config("spark.sql.warehouse.dir", warehouse).config(org.apache.carbondata.core.constants.CarbonCommonConstants.STORE_LOCATION, storeLocation).getOrCreateCarbonSession(storeLocation, metaStoreDB)
But when I tried to create a table, I got an error:
scala> carbon.sql("create table hive_carbon(id int, name string, scale decimal, country string, salary double) STORED BY 'carbondata'")
Hive Session ID = 9542367b-a1d6-4ee0-8021-3db9d99fbd28
java.lang.NoSuchMethodError: org.apache.spark.sql.hive.HiveSessionResourceLoader.<init>(Lorg/apache/spark/sql/SparkSession;Lorg/apache/spark/sql/hive/client/HiveClient;)V
at org.apache.spark.sql.hive.CarbonSessionStateBuilder.resourceLoader$lzycompute(CarbonSessionStateBuilder.scala:202)
at org.apache.spark.sql.hive.CarbonSessionStateBuilder.resourceLoader(CarbonSessionStateBuilder.scala:200)
at org.apache.spark.sql.hive.CarbonSessionStateBuilder.catalog$lzycompute(CarbonSessionStateBuilder.scala:189)
at org.apache.spark.sql.hive.CarbonSessionStateBuilder.catalog(CarbonSessionStateBuilder.scala:180)
at org.apache.spark.sql.hive.CarbonSessionStateBuilder.analyzer(CarbonSessionStateBuilder.scala:206)
at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anonfun$build$2.apply(BaseSessionStateBuilder.scala:293)
at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anonfun$build$2.apply(BaseSessionStateBuilder.scala:293)
at org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:79)
at org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:79)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
at org.apache.spark.sql.CarbonSession.withProfiler(CarbonSession.scala:127)
at org.apache.spark.sql.CarbonSession.sql(CarbonSession.scala:86)
... 51 elided
I checked all maven artifacts, they have same versions.
Maybe someone can help me to fight with this error?
Thank you for help!
Best regards,
Vasily
|
Locked
