problem with branch-1.1

[孙而焓]

hello，
        I tried to use branch-1.1 under hdp2.6.0-spark2.1.0 and met so many problems.
        1.build: use "mvn package -DskipTests -Pspark-2.1 -Dspark.version=2.1.0 -Phadoop-2.7.2" and
        while compiling core module,throw class not found error,like org.apache.thrift.TBase,
        modify pom.xml in core and add libthrift dependency,compile successfully.
        2.running in spark-shell like ;  
            spark-shell --jars carbondata_2.11-1.1.1-SNAPSHOT-shade-hadoop2.7.2.jar
            scala>import org.apache.spark.sql.SparkSession
            scala>import org.apache.spark.sql.CarbonSession._
            scala>val cc = SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(hdfs path)
            and got error:
            java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/CatalystConf
at org.apache.spark.sql.hive.CarbonSessionState.analyzer$lzycompute(CarbonSessionState.scala:127)
at org.apache.spark.sql.hive.CarbonSessionState.analyzer(CarbonSessionState.scala:126)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
... 52 elided
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.CatalystConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 59 more
check spark-catalyst.jar and didn't found class CatalystConf
3.use apache-spark catalyst jars and rerun with
spark-shell --jars carbondata_2.11-1.1.1-SNAPSHOT-shade-hadoop2.7.2.jar,spark-catalyst_2.11-2.1.0.jar
and got error message like:
java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/CatalystConf
at org.apache.spark.sql.hive.CarbonSessionState.analyzer$lzycompute(CarbonSessionState.scala:127)
at org.apache.spark.sql.hive.CarbonSessionState.analyzer(CarbonSessionState.scala:126)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
... 52 elided
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.CatalystConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 59 more



[hidden email]

[Erlu Chen]

Hi,

Please try mvn package -DskipTests -Pspark-2.1 -Dspark.version=2.1.0 -Phadoop-2.7.2 with hadoop2.7.2 and spark 2.

I have just tested, it ok to compile.

[INFO] Reactor Summary:
[INFO]
[INFO] Apache CarbonData :: Parent ........................ SUCCESS [  1.657 s]
[INFO] Apache CarbonData :: Common ........................ SUCCESS [  1.870 s]
[INFO] Apache CarbonData :: Core .......................... SUCCESS [ 25.003 s]
[INFO] Apache CarbonData :: Processing .................... SUCCESS [  1.941 s]
[INFO] Apache CarbonData :: Hadoop ........................ SUCCESS [  2.017 s]
[INFO] Apache CarbonData :: Spark Common .................. SUCCESS [ 20.622 s]
[INFO] Apache CarbonData :: Spark2 ........................ SUCCESS [ 39.956 s]
[INFO] Apache CarbonData :: Spark Common Test ............. SUCCESS [  4.024 s]
[INFO] Apache CarbonData :: Assembly ...................... SUCCESS [  3.400 s]
[INFO] Apache CarbonData :: Spark2 Examples ............... SUCCESS [  9.718 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:50 min
[INFO] Finished at: 2017-06-23T17:55:28+08:00
[INFO] Final Memory: 83M/860M
[INFO] ------------------------------------------------------------------------
bogon:carbondata erlu$ git branch
* branch-1.1

Regrads.
Chenerlu

[David CaiQiang]

The spark core version of  hdp2.6.0-spark2.1.0 is spark 2.1.1.
In spark 2.1.1, CatalystConf was already removed.

We raised PR to support it and will merge it at later.
https://github.com/apache/carbondata/pull/1096
https://github.com/apache/carbondata/pull/1017

And the command will be "mvn package -DskipTests -Pspark-2.1 -Dspark.version=2.1.1 -Phadoop-2.7.2"

[孙而焓]

delete aliyun mirror from maven by editing settings.xml file and remove scala-2.11 from idea's global library even you assigned scala-2.10.

[Erlu Chen]

The key point is the version of spark and carbondata should match.

Regards.
Chenerlu.