Disscusion shall CI support run carbondata based on multi version spark?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Disscusion shall CI support run carbondata based on multi version spark?

zhujin
One issue:
I modified the spark.version in pom.xml,using spark1.6.2, then compliation failed.


Root cause:
There was a "unused import statement" warinng in CarbonOptimizer class before, we imported AggregationExpression like the following :
import org.apache.spark.sql.catalyst.expressions.aggregate._
import org.apache.spark.sql.catalyst.expressions._
But in spark1.6.2, AggregateExpressions is moved to subpackage "aggregate" from "expressions"(that is for spark1.5.2).
So if we didn't known this change, we removed this import "import org.apache.spark.sql.catalyst.expressions.aggregate._", it will cause compliation failure when using spark1.6.2


Question:
So, maybe the CI should verify carbondata based on different version spark, then it will be more helful to check the correctness of the commits, shall we?
Reply | Threaded
Open this post in threaded view
|

Re: Disscusion shall CI support run carbondata based on multi version spark?

Liang Chen
Administrator

Yes, need to solve it , the CI should support different spark version.

Regards
Liang

zhujin wrote
One issue:
I modified the spark.version in pom.xml,using spark1.6.2, then compliation failed.


Root cause:
There was a "unused import statement" warinng in CarbonOptimizer class before, we imported AggregationExpression like the following :
import org.apache.spark.sql.catalyst.expressions.aggregate._
import org.apache.spark.sql.catalyst.expressions._
But in spark1.6.2, AggregateExpressions is moved to subpackage "aggregate" from "expressions"(that is for spark1.5.2).
So if we didn't known this change, we removed this import "import org.apache.spark.sql.catalyst.expressions.aggregate._", it will cause compliation failure when using spark1.6.2


Question:
So, maybe the CI should verify carbondata based on different version spark, then it will be more helful to check the correctness of the commits, shall we?
Reply | Threaded
Open this post in threaded view
|

Re: Disscusion shall CI support run carbondata based on multi version spark?

Vimal Das Kammath
Yes, I Agree. CI should be configured to build Carbon on different spark
versions.

On Fri, Oct 14, 2016 at 7:56 AM, Liang Chen <[hidden email]> wrote:

>
> Yes, need to solve it , the CI should support different spark version.
>
> Regards
> Liang
>
>
> zhujin wrote
> > One issue:
> > I modified the spark.version in pom.xml,using spark1.6.2, then
> compliation
> > failed.
> >
> >
> > Root cause:
> > There was a "unused import statement" warinng in CarbonOptimizer class
> > before, we imported AggregationExpression like the following :
> > import org.apache.spark.sql.catalyst.expressions.aggregate._
> > import org.apache.spark.sql.catalyst.expressions._
> > But in spark1.6.2, AggregateExpressions is moved to subpackage
> "aggregate"
> > from "expressions"(that is for spark1.5.2).
> > So if we didn't known this change, we removed this import "import
> > org.apache.spark.sql.catalyst.expressions.aggregate._", it will cause
> > compliation failure when using spark1.6.2
> >
> >
> > Question:
> > So, maybe the CI should verify carbondata based on different version
> > spark, then it will be more helful to check the correctness of the
> > commits, shall we?
>
>
>
>
>
> --
> View this message in context: http://apache-carbondata-
> mailing-list-archive.1130556.n5.nabble.com/Disscusion-
> shall-CI-support-run-carbondata-based-on-multi-
> version-spark-tp1836p1890.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive
> at Nabble.com.
>