problem with branch-1.1

Posted by 孙而焓 on Jun 23, 2017; 8:49am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/problem-with-branch-1-1-tp16004.html

hello，
        I tried to use branch-1.1 under hdp2.6.0-spark2.1.0 and met so many problems.
        1.build: use "mvn package -DskipTests -Pspark-2.1 -Dspark.version=2.1.0 -Phadoop-2.7.2" and
        while compiling core module,throw class not found error,like org.apache.thrift.TBase,
        modify pom.xml in core and add libthrift dependency,compile successfully.
        2.running in spark-shell like ;  
            spark-shell --jars carbondata_2.11-1.1.1-SNAPSHOT-shade-hadoop2.7.2.jar
            scala>import org.apache.spark.sql.SparkSession
            scala>import org.apache.spark.sql.CarbonSession._
            scala>val cc = SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(hdfs path)
            and got error:
            java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/CatalystConf
at org.apache.spark.sql.hive.CarbonSessionState.analyzer$lzycompute(CarbonSessionState.scala:127)
at org.apache.spark.sql.hive.CarbonSessionState.analyzer(CarbonSessionState.scala:126)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
... 52 elided
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.CatalystConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 59 more
check spark-catalyst.jar and didn't found class CatalystConf
3.use apache-spark catalyst jars and rerun with
spark-shell --jars carbondata_2.11-1.1.1-SNAPSHOT-shade-hadoop2.7.2.jar,spark-catalyst_2.11-2.1.0.jar
and got error message like:
java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/CatalystConf
at org.apache.spark.sql.hive.CarbonSessionState.analyzer$lzycompute(CarbonSessionState.scala:127)
at org.apache.spark.sql.hive.CarbonSessionState.analyzer(CarbonSessionState.scala:126)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
... 52 elided
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.CatalystConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 59 more



[hidden email]