[GitHub] carbondata pull request #964: [CARBONDATA-1099] Fixed bug for carbon-spark-s...

[qiuchenjian-2]

GitHub user xuchuanyin opened a pull request:

    https://github.com/apache/carbondata/pull/964

    [CARBONDATA-1099] Fixed bug for carbon-spark-shell in spark2 environment

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
   
     - [ ] Make sure the PR title is formatted like:
       `[CARBONDATA-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [ ] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
     - [ ] Testing done
     
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - What manual testing you have done?
            - Any additional information to help reviewers in testing this change.
             
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
                     
    ---


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xuchuanyin/carbondata update_shell

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/964.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #964
   
----
commit 7bf23f87c9afd24eb3b711a6ff51bc2339aa3e9a
Author: xuchuanyin <[hidden email]>
Date:   2017-05-27T08:49:12Z

    Update carbon-spark-shell
   
    add search for scala-2.11 folder when compil spark2

commit 3752c2256ef23b66a1a99a89a44db71d9d455c35
Author: xuchuanyin <[hidden email]>
Date:   2017-05-27T08:52:44Z

    Update CarbonSparkILoop.scala

commit fc294bb9204c22ea19b7586e20a1d45812c81915
Author: xuchuanyin <[hidden email]>
Date:   2017-05-27T08:54:02Z

    Create Main.scala

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/964
 
    retest


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/964
 
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/964
 
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user chenliang613 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/964#discussion_r119642951
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java ---
    @@ -43,7 +43,7 @@
        * Attribute for Carbon LOGGER
        */
       private static final LogService LOGGER =
    -      LogServiceFactory.getLogService(ForwardDictionaryCache.class.getName());
    +      LogServiceFactory.getLogService(ReverseDictionaryCache.class.getName());
    --- End diff --
   
    why need to change this code  for fix carbon-spark-shell issues?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/964#discussion_r119761693
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java ---
    @@ -43,7 +43,7 @@
        * Attribute for Carbon LOGGER
        */
       private static final LogService LOGGER =
    -      LogServiceFactory.getLogService(ForwardDictionaryCache.class.getName());
    +      LogServiceFactory.getLogService(ReverseDictionaryCache.class.getName());
    --- End diff --
   
    yeah, there is no need to fix this in the current issue. I just found it when I submitted the code...
   
    so, should I rollback this change and start a new issue, OR just add another comment in this issue about this change?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user chenliang613 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/964#discussion_r120256136
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java ---
    @@ -43,7 +43,7 @@
        * Attribute for Carbon LOGGER
        */
       private static final LogService LOGGER =
    -      LogServiceFactory.getLogService(ForwardDictionaryCache.class.getName());
    +      LogServiceFactory.getLogService(ReverseDictionaryCache.class.getName());
    --- End diff --
   
    i suggest that don't need to change this part in this PR.  it would be better that each pr clearly fix each issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/964#discussion_r120272132
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java ---
    @@ -43,7 +43,7 @@
        * Attribute for Carbon LOGGER
        */
       private static final LogService LOGGER =
    -      LogServiceFactory.getLogService(ForwardDictionaryCache.class.getName());
    +      LogServiceFactory.getLogService(ReverseDictionaryCache.class.getName());
    --- End diff --
   
    OK, I've already rollback this


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/964
 
   
    Refer to this link for build results (access rights to CI server needed):
    https://builds.apache.org/job/carbondata-pr-spark-1.6/90/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/964
 
    retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/964
 
   
    Refer to this link for build results (access rights to CI server needed):
    https://builds.apache.org/job/carbondata-pr-spark-1.6/96/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user chenerlu commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/964#discussion_r120403537
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/repl/CarbonSparkILoop.scala ---
    @@ -19,54 +19,66 @@ package org.apache.spark.repl
     
     class CarbonSparkILoop extends SparkILoop {
     
    -  override def initializeSpark() {
    -    intp.beQuietDuring {
    -      command("""
    -         if(org.apache.spark.repl.carbon.Main.interp == null) {
    -           org.apache.spark.repl.carbon.Main.main(Array[String]())
    -         }
    -              """)
    -      command("val i1 = org.apache.spark.repl.carbon.Main.interp")
    -      command("import i1._")
    -      command("""
    -         @transient val sc = {
    -           val _sc = i1.createSparkContext()
    -           println("Spark context available as sc.")
    -           _sc
    -         }
    -              """)
    -      command("import org.apache.spark.SparkContext._")
    -      command("import org.apache.spark.sql.CarbonContext")
    -      command("""
    -         @transient val cc = {
    -           val _cc = {
    -             import java.io.File
    -             val path = System.getenv("CARBON_HOME") + "/bin/carbonshellstore"
    -             val store = new File(path)
    -             store.mkdirs()
    -             val storePath = sc.getConf.getOption("spark.carbon.storepath")
    -                  .getOrElse(store.getCanonicalPath)
    -             new CarbonContext(sc, storePath, store.getCanonicalPath)
    -           }
    -           println("Carbon context available as cc.")
    -           _cc
    -         }
    -              """)
    +  private def initOriginSpark(): Unit = {
    +    processLine("""
    +        @transient val spark = if (org.apache.spark.repl.Main.sparkSession != null) {
    +            org.apache.spark.repl.Main.sparkSession
    +          } else {
    +            org.apache.spark.repl.Main.createSparkSession()
    +          }
    +        @transient val sc = {
    +          val _sc = spark.sparkContext
    +          if (_sc.getConf.getBoolean("spark.ui.reverseProxy", false)) {
    +            val proxyUrl = _sc.getConf.get("spark.ui.reverseProxyUrl", null)
    +            if (proxyUrl != null) {
    +              println(s"Spark Context Web UI is available at " +
    +                s"${proxyUrl}/proxy/${_sc.applicationId}")
    +            } else {
    +              println(s"Spark Context Web UI is available at Spark Master Public URL")
    +            }
    +          } else {
    +            _sc.uiWebUrl.foreach {
    +              webUrl => println(s"Spark context Web UI available at ${webUrl}")
    +            }
    +          }
    +          println("Spark context available as 'sc' " +
    +            s"(master = ${_sc.master}, app id = ${_sc.applicationId}).")
    +          println("Spark session available as 'spark'.")
    +          _sc
    +        }
    +        """)
    +    processLine("import org.apache.spark.SparkContext._")
    +    processLine("import spark.implicits._")
    +    processLine("import spark.sql")
    +    processLine("import org.apache.spark.sql.functions._")
    +  }
     
    -      command("import org.apache.spark.sql.SQLContext")
    -      command("""
    -         @transient val sqlContext = {
    -           val _sqlContext = new SQLContext(sc)
    -           println("SQL context available as sqlContext.")
    -           _sqlContext
    -         }
    -              """)
    -      command("import sqlContext.implicits._")
    -      command("import sqlContext.sql")
    +  private def initCarbon(): Unit = {
    +    processLine("""
    +      import org.apache.spark.sql.SparkSession
    +      import org.apache.spark.sql.CarbonSession._
    +      @transient val carbon = {
    +        val _carbon = {
    +          import java.io.File
    +          val path = System.getenv("CARBON_HOME") + "/bin/carbonshellstore"
    +          val store = new File(path)
    +          store.mkdirs()
    +          val storePath = sc.getConf.getOption("spark.carbon.storepath")
    +                 .getOrElse(store.getCanonicalPath)
    +          SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(storePath)
    --- End diff --
   
    If you do not specify the metastore path, it will generate carbon.metastore which is in same level with carbondata project.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/964#discussion_r120515436
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/repl/CarbonSparkILoop.scala ---
    @@ -19,54 +19,66 @@ package org.apache.spark.repl
     
     class CarbonSparkILoop extends SparkILoop {
     
    -  override def initializeSpark() {
    -    intp.beQuietDuring {
    -      command("""
    -         if(org.apache.spark.repl.carbon.Main.interp == null) {
    -           org.apache.spark.repl.carbon.Main.main(Array[String]())
    -         }
    -              """)
    -      command("val i1 = org.apache.spark.repl.carbon.Main.interp")
    -      command("import i1._")
    -      command("""
    -         @transient val sc = {
    -           val _sc = i1.createSparkContext()
    -           println("Spark context available as sc.")
    -           _sc
    -         }
    -              """)
    -      command("import org.apache.spark.SparkContext._")
    -      command("import org.apache.spark.sql.CarbonContext")
    -      command("""
    -         @transient val cc = {
    -           val _cc = {
    -             import java.io.File
    -             val path = System.getenv("CARBON_HOME") + "/bin/carbonshellstore"
    -             val store = new File(path)
    -             store.mkdirs()
    -             val storePath = sc.getConf.getOption("spark.carbon.storepath")
    -                  .getOrElse(store.getCanonicalPath)
    -             new CarbonContext(sc, storePath, store.getCanonicalPath)
    -           }
    -           println("Carbon context available as cc.")
    -           _cc
    -         }
    -              """)
    +  private def initOriginSpark(): Unit = {
    +    processLine("""
    +        @transient val spark = if (org.apache.spark.repl.Main.sparkSession != null) {
    +            org.apache.spark.repl.Main.sparkSession
    +          } else {
    +            org.apache.spark.repl.Main.createSparkSession()
    +          }
    +        @transient val sc = {
    +          val _sc = spark.sparkContext
    +          if (_sc.getConf.getBoolean("spark.ui.reverseProxy", false)) {
    +            val proxyUrl = _sc.getConf.get("spark.ui.reverseProxyUrl", null)
    +            if (proxyUrl != null) {
    +              println(s"Spark Context Web UI is available at " +
    +                s"${proxyUrl}/proxy/${_sc.applicationId}")
    +            } else {
    +              println(s"Spark Context Web UI is available at Spark Master Public URL")
    +            }
    +          } else {
    +            _sc.uiWebUrl.foreach {
    +              webUrl => println(s"Spark context Web UI available at ${webUrl}")
    +            }
    +          }
    +          println("Spark context available as 'sc' " +
    +            s"(master = ${_sc.master}, app id = ${_sc.applicationId}).")
    +          println("Spark session available as 'spark'.")
    +          _sc
    +        }
    +        """)
    +    processLine("import org.apache.spark.SparkContext._")
    +    processLine("import spark.implicits._")
    +    processLine("import spark.sql")
    +    processLine("import org.apache.spark.sql.functions._")
    +  }
     
    -      command("import org.apache.spark.sql.SQLContext")
    -      command("""
    -         @transient val sqlContext = {
    -           val _sqlContext = new SQLContext(sc)
    -           println("SQL context available as sqlContext.")
    -           _sqlContext
    -         }
    -              """)
    -      command("import sqlContext.implicits._")
    -      command("import sqlContext.sql")
    +  private def initCarbon(): Unit = {
    +    processLine("""
    +      import org.apache.spark.sql.SparkSession
    +      import org.apache.spark.sql.CarbonSession._
    +      @transient val carbon = {
    +        val _carbon = {
    +          import java.io.File
    +          val path = System.getenv("CARBON_HOME") + "/bin/carbonshellstore"
    +          val store = new File(path)
    +          store.mkdirs()
    +          val storePath = sc.getConf.getOption("spark.carbon.storepath")
    +                 .getOrElse(store.getCanonicalPath)
    +          SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(storePath)
    --- End diff --
   
    Yes, It works just the same as that in integration/spark1.
   
    Please refer to : https://github.com/apache/carbondata/blob/master/integration/spark/src/main/scala/org/apache/spark/repl/CarbonSparkILoop.scala#L47


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/964
 
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/964
 
    carbon-spark-shell has been removed in our discussion in maillist，so we can close this pr.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user xuchuanyin closed the pull request at:

    https://github.com/apache/carbondata/pull/964


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---