Support for spark 2.1.1 (HDP spark versions)

[satish.sidnakoppa]

<http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t389/versions.png>

The carbon supported spark versions and Hortonworks - HDP supported versions
are different.
I am unable to integrate carbondata and HDP .

In brief , how can I build carbondata jar to on spark 2.1.1
Because I am working on HDP 2.6.2 that has spark 2.1.1 as MAJOR version
whereas carbondata default support is 2.1.0

Due to this when I run carbon session creation.I get below error

val cc =
SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(hdfs
path)
            and got error:
            java.lang.NoClassDefFoundError:
org/apache/spark/sql/catalyst/CatalystConf



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[Liang Chen]

Hi

Did you successfully compile with "mvn -DskipTests -Pspark-2.1
-Dspark.version=2.1.1 clean package" ?
I used the community spark 2.1.1, it is working fine.

Regards
Liang


satish.sidnakoppa wrote

> &lt;http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t389/versions.png&gt; 
>
> The carbon supported spark versions and Hortonworks - HDP supported
> versions
> are different.
> I am unable to integrate carbondata and HDP .
>
> In brief , how can I build carbondata jar to on spark 2.1.1
> Because I am working on HDP 2.6.2 that has spark 2.1.1 as MAJOR version
> whereas carbondata default support is 2.1.0
>
> Due to this when I run carbon session creation.I get below error
>
> val cc =
> SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(hdfs
> path)
>             and got error:
>             java.lang.NoClassDefFoundError:
> org/apache/spark/sql/catalyst/CatalystConf
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[satish.sidnakoppa]

I tried to build using
mvn -DskipTests -Pspark-2.1 -Dspark.version=2.1.1 clean package

Build failing for spark 2.1.1


[INFO] Apache CarbonData :: Parent ........................ SUCCESS [  3.981
s]
.
.
.
[INFO] Apache CarbonData :: Bloom Index DataMap ........... SUCCESS [ 13.166
s]
[INFO] Apache CarbonData :: Spark2 ........................ FAILURE [ 27.384
s]

ERROR] Failed to execute goal
org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project
carbondata-spark2: wrap: org.apache.commons.exec.ExecuteException: Process
exited with an error: 1(Exit value: 1) -> [Help 1]


But the build is failing




-----
regards
Satish
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/