[GitHub] incubator-carbondata pull request #417: fix conf issue for scanrdd

[qiuchenjian-2]

GitHub user scwf opened a pull request:

    https://github.com/apache/incubator-carbondata/pull/417

    fix conf issue for scanrdd

    we should sync the conf in executor side

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/KirinKing/incubator-carbondata fix-scan

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/417.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #417
   
----
commit 1c54cd4ef942be0f5e2eef40c34db98a2233a0dc
Author: wangfei <[hidden email]>
Date:   2016-12-10T23:24:03Z

    fix conf issue for scanrdd

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user scwf commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/417
 
    /cc @QiangCai @jackylk plz review this


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/417
 
    Build Failed  with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/90/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/417
 
    Build Failed  with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/91/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/417
 
    Build Failed  with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/92/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/417
 
    Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/93/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/417
 
    Add to whitelist


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/417
 
    Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/96/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/417#discussion_r91847537
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
    @@ -37,6 +36,9 @@ object CarbonEnv {
     
       @volatile private var carbonEnv: CarbonEnv = _
     
    +  // set readsupport class global so that the executor can get it.
    +  SparkReadSupport.readSupportClass = classOf[SparkRowReadSupportImpl]
    --- End diff --
   
    move it to init function


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/417#discussion_r91847539
 
    --- Diff: integration/spark/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
    @@ -21,22 +21,24 @@ import org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
     import org.apache.spark.sql.hive.CarbonMetastore
     
     import org.apache.carbondata.hadoop.readsupport.impl.RawDataReadSupport
    -import org.apache.carbondata.spark.rdd.SparkCommonEnv
    +import org.apache.carbondata.spark.rdd.SparkReadSupport
     
     case class CarbonEnv(carbonMetastore: CarbonMetastore)
     
     object CarbonEnv {
     
       @volatile private var carbonEnv: CarbonEnv = _
     
    +  // set readsupport class global so that the executor can get it.
    +  SparkReadSupport.readSupportClass = classOf[RawDataReadSupport]
    --- End diff --
   
    move it to init function


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user scwf commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/417#discussion_r91848570
 
    --- Diff: integration/spark/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
    @@ -21,22 +21,24 @@ import org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
     import org.apache.spark.sql.hive.CarbonMetastore
     
     import org.apache.carbondata.hadoop.readsupport.impl.RawDataReadSupport
    -import org.apache.carbondata.spark.rdd.SparkCommonEnv
    +import org.apache.carbondata.spark.rdd.SparkReadSupport
     
     case class CarbonEnv(carbonMetastore: CarbonMetastore)
     
     object CarbonEnv {
     
       @volatile private var carbonEnv: CarbonEnv = _
     
    +  // set readsupport class global so that the executor can get it.
    +  SparkReadSupport.readSupportClass = classOf[RawDataReadSupport]
    --- End diff --
   
    No, this is the key point, init is called in driver, see the comment, place here is to let executor get the read support class


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user scwf commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/417#discussion_r91848571
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
    @@ -37,6 +36,9 @@ object CarbonEnv {
     
       @volatile private var carbonEnv: CarbonEnv = _
     
    +  // set readsupport class global so that the executor can get it.
    +  SparkReadSupport.readSupportClass = classOf[SparkRowReadSupportImpl]
    --- End diff --
   
    also the same


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/417#discussion_r91849094
 
    --- Diff: integration/spark/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
    @@ -21,22 +21,24 @@ import org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
     import org.apache.spark.sql.hive.CarbonMetastore
     
     import org.apache.carbondata.hadoop.readsupport.impl.RawDataReadSupport
    -import org.apache.carbondata.spark.rdd.SparkCommonEnv
    +import org.apache.carbondata.spark.rdd.SparkReadSupport
     
     case class CarbonEnv(carbonMetastore: CarbonMetastore)
     
     object CarbonEnv {
     
       @volatile private var carbonEnv: CarbonEnv = _
     
    +  // set readsupport class global so that the executor can get it.
    +  SparkReadSupport.readSupportClass = classOf[RawDataReadSupport]
    --- End diff --
   
    ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/417#discussion_r91849096
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
    @@ -37,6 +36,9 @@ object CarbonEnv {
     
       @volatile private var carbonEnv: CarbonEnv = _
     
    +  // set readsupport class global so that the executor can get it.
    +  SparkReadSupport.readSupportClass = classOf[SparkRowReadSupportImpl]
    --- End diff --
   
    ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-carbondata/pull/417


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---