[GitHub] incubator-carbondata pull request #415: Fix integrate with spark 2.1

[qiuchenjian-2]

GitHub user scwf opened a pull request:

    https://github.com/apache/incubator-carbondata/pull/415

    Fix integrate with spark 2.1

   

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/KirinKing/incubator-carbondata fix-2.1

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/415.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #415
   
----
commit 74e9b98d2596d5c52fa8ef43aeddeaf6ca689c01
Author: wangfei <[hidden email]>
Date:   2016-12-09T09:28:59Z

    use carbon property to get the store path

commit 906ab88ed4ebe178bf1a52e1bb26f3dc12d4f07c
Author: wangfei <[hidden email]>
Date:   2016-12-09T09:51:32Z

    remove nouse imports

commit 14cfe617f46e84a27f83da2fe2edd1e73298b7d1
Author: wangfei <[hidden email]>
Date:   2016-12-09T13:14:04Z

    fix bugs with spark 2.1

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user scwf commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/415
 
    /cc @QiangCai  can you try this fix


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/415
 
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/415
 
    Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/88/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/415
 
    Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/89/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/415
 
    Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/94/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/415
 
    Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/95/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/415
 
    Add to whitelist


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/415#discussion_r91847495
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
    @@ -223,10 +223,11 @@ case class CarbonDictionaryDecoder(
     
     
     
    -class CarbonDecoderRDD(relations: Seq[CarbonDecoderRelation],
    -                profile: CarbonProfile,
    -                aliasMap: CarbonAliasDecoderRelation,
    -                prev: RDD[Row],
    +class CarbonDecoderRDD(
    +    relations: Seq[CarbonDecoderRelation],
    +    profile: CarbonProfile,
    +    aliasMap: CarbonAliasDecoderRelation,
    +    prev: RDD[Row],
                            output: Seq[Attribute])
    --- End diff --
   
    incorrect indentation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/415#discussion_r91847501
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/CarbonLateDecodeStrategy.scala ---
    @@ -101,12 +103,11 @@ private[sql] class CarbonLateDecodeStrategy extends SparkStrategy {
       }
     
       protected def pruneFilterProject(
    -                                    relation: LogicalRelation,
    -                                    projects: Seq[NamedExpression],
    -                                    filterPredicates: Seq[Expression],
    -                                    scanBuilder: (Seq[Attribute], Array[Filter],
    -                                        ArrayBuffer[AttributeReference]) =>
    -                                        RDD[InternalRow]) = {
    +      relation: LogicalRelation,
    +      projects: Seq[NamedExpression],
    +      filterPredicates: Seq[Expression],
    +      scanBuilder: (Seq[Attribute], Array[Filter],
    +      ArrayBuffer[AttributeReference]) => RDD[InternalRow]) = {
    --- End diff --
   
    incorrect indentation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/415#discussion_r91847518
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/CarbonLateDecodeStrategy.scala ---
    @@ -212,20 +212,20 @@ private[sql] class CarbonLateDecodeStrategy extends SparkStrategy {
             }
             attr
           }
    -      val scan = execution.DataSourceScanExec.create(
    +      val scan = new execution.RowDataSourceScanExec(
             updateProject,
             scanBuilder(updateRequestedColumns, candidatePredicates, pushedFilters, needDecoder),
    -        relation.relation, metadata, relation.metastoreTableIdentifier)
    +        relation.relation, UnknownPartitioning(9), metadata, None)
    --- End diff --
   
    why is it 9? give comment for this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/415#discussion_r91847522
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/CarbonLateDecodeStrategy.scala ---
    @@ -212,20 +212,20 @@ private[sql] class CarbonLateDecodeStrategy extends SparkStrategy {
             }
             attr
           }
    -      val scan = execution.DataSourceScanExec.create(
    +      val scan = new execution.RowDataSourceScanExec(
             updateProject,
             scanBuilder(updateRequestedColumns, candidatePredicates, pushedFilters, needDecoder),
    -        relation.relation, metadata, relation.metastoreTableIdentifier)
    +        relation.relation, UnknownPartitioning(9), metadata, None)
           filterCondition.map(execution.FilterExec(_, scan)).getOrElse(scan)
         } else {
           // Don't request columns that are only referenced by pushed filters.
           val requestedColumns =
           (projectSet ++ filterSet -- handledSet).map(relation.attributeMap).toSeq
           val updateRequestedColumns = updateRequestedColumnsFunc(requestedColumns, table, needDecoder)
    -      val scan = execution.DataSourceScanExec.create(
    +      val scan = new execution.RowDataSourceScanExec(
             updateRequestedColumns,
             scanBuilder(updateRequestedColumns, candidatePredicates, pushedFilters, needDecoder),
    -        relation.relation, metadata, relation.metastoreTableIdentifier)
    +        relation.relation, UnknownPartitioning(0), metadata, None)
    --- End diff --
   
    give comment for 0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/415
 
    Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/99/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-carbondata/pull/415


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---