[GitHub] incubator-carbondata pull request #606: [CARBONDATA-713] Make the store path...

[qiuchenjian-2]

GitHub user watermen opened a pull request:

    https://github.com/apache/incubator-carbondata/pull/606

    [CARBONDATA-713] Make the store path in right priority

    The store location of carbon comes from three places:
    * default location path in code(../carbon.store)
    * configurate "carbon.storelocation" in carbon.properties
    * pass the location as the parameter
   
    The priority is low to high.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/watermen/incubator-carbondata CARBONDATA-713

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/606.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #606
   
----
commit 7b4c2dbf0262d20e894b6820533e63f8ef2a0878
Author: Yadong Qi <[hidden email]>
Date:   2017-02-21T03:01:06Z

    Make the store path in right priority.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/606
 
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user watermen commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/606
 
    cc @chenliang613 @jackylk


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/606
 
    ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/606
 
    Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/929/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/606#discussion_r102356794
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/spark/util/FileUtils.scala ---
    @@ -91,4 +94,24 @@ object FileUtils {
         }
       }
     
    +  /*
    +   * The store location of carbon comes from three places:
    +   * 1. default location path in code(../carbon.store)
    +   * 2. configurate "carbon.storelocation" in carbon.properties
    +   * 3. pass the location as the parameter
    +   * The priority is low to high.
    +   */
    +  def getHighestPriorityStorePath(storePathParameter: String): String = {
    +    val storePathDefaultValue =
    +      new File(CarbonCommonConstants.STORE_LOCATION_DEFAULT_VAL).getCanonicalPath
    +    val storePathProperties = CarbonProperties.getInstance()
    +      .getProperty(CarbonCommonConstants.STORE_LOCATION)
    +    if (!storePathParameter.equals(storePathDefaultValue)) {
    --- End diff --
   
    This is strange to me, can you change the constructor of CarbonContext and CarbonSession, to make storePath as an optional parameter whose default value is None. Then you can check weather it is None here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user watermen commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/606#discussion_r102370250
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/spark/util/FileUtils.scala ---
    @@ -91,4 +94,24 @@ object FileUtils {
         }
       }
     
    +  /*
    +   * The store location of carbon comes from three places:
    +   * 1. default location path in code(../carbon.store)
    +   * 2. configurate "carbon.storelocation" in carbon.properties
    +   * 3. pass the location as the parameter
    +   * The priority is low to high.
    +   */
    +  def getHighestPriorityStorePath(storePathParameter: String): String = {
    +    val storePathDefaultValue =
    +      new File(CarbonCommonConstants.STORE_LOCATION_DEFAULT_VAL).getCanonicalPath
    +    val storePathProperties = CarbonProperties.getInstance()
    +      .getProperty(CarbonCommonConstants.STORE_LOCATION)
    +    if (!storePathParameter.equals(storePathDefaultValue)) {
    --- End diff --
   
    @jackylk My first idea is like yours, but I worry about it will change the API. If it can be changed, I will do this solution.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/606#discussion_r102372504
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/spark/util/FileUtils.scala ---
    @@ -91,4 +94,24 @@ object FileUtils {
         }
       }
     
    +  /*
    +   * The store location of carbon comes from three places:
    +   * 1. default location path in code(../carbon.store)
    +   * 2. configurate "carbon.storelocation" in carbon.properties
    +   * 3. pass the location as the parameter
    +   * The priority is low to high.
    +   */
    +  def getHighestPriorityStorePath(storePathParameter: String): String = {
    +    val storePathDefaultValue =
    +      new File(CarbonCommonConstants.STORE_LOCATION_DEFAULT_VAL).getCanonicalPath
    +    val storePathProperties = CarbonProperties.getInstance()
    +      .getProperty(CarbonCommonConstants.STORE_LOCATION)
    +    if (!storePathParameter.equals(storePathDefaultValue)) {
    --- End diff --
   
    I think user can still use `val cc = new CarbonContext(sc)`, so API will not change
    @ravipesala what is your opinion


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user watermen commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/606#discussion_r102375559
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/spark/util/FileUtils.scala ---
    @@ -91,4 +94,24 @@ object FileUtils {
         }
       }
     
    +  /*
    +   * The store location of carbon comes from three places:
    +   * 1. default location path in code(../carbon.store)
    +   * 2. configurate "carbon.storelocation" in carbon.properties
    +   * 3. pass the location as the parameter
    +   * The priority is low to high.
    +   */
    +  def getHighestPriorityStorePath(storePathParameter: String): String = {
    +    val storePathDefaultValue =
    +      new File(CarbonCommonConstants.STORE_LOCATION_DEFAULT_VAL).getCanonicalPath
    +    val storePathProperties = CarbonProperties.getInstance()
    +      .getProperty(CarbonCommonConstants.STORE_LOCATION)
    +    if (!storePathParameter.equals(storePathDefaultValue)) {
    --- End diff --
   
    @jackylk I will keep all of constructors API(`CarbonContext(sc)`, `CarbonContext(sc, storePath)`, `CarbonContext(sc, storePath, metaStorePath)`).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/606
 
    @watermen @jackylk we should always give high priority to path provided in carbon.properties.  All these issues are because of the paths provided through context and carbon.properties are different.
    In cluster we always takes the path from carbon.properties, not from context.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/606
 
    Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/930/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user chenliang613 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/606
 
    @ravipesala  as we discussed on yesterday,  when user doesn't specify store location(HDFS) in CarbonContext, system if can directly get store location from carbon.properties for keeping consistent,or not ?
    ----------------------------------------
    All these issues are because of the paths provided through context and carbon.properties are different.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user Hexiaoqiao commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/606
 
    @chenliang613 @ravipesala it seems not get the item `carbon.storelocation` from carbon.properties if create CarbonContext with `val cc = new CarbonContext(sc)` and always use `./carbon.store` as the secondary parameter when run on cluster mode currently, the detailed exception as maillist mentioned:
    http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/template/NamlServlet.jtp?macro=raw_mail&node=7707
    pls correct me if i am wrong.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user chenliang613 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/606
 
    @Hexiaoqiao  Yes, you are right, currently do like this what you mentioned above.
   
    I just proposed one solution about how to keep store location consistent between carboncontext and carbon.properties.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user watermen commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/606
 
    @ravipesala Can you tell me where is the code about `In cluster we always takes the path from carbon.properties, not from context.`? I only found in SparkEnv.init, we take the path from context.
    ```scala
      def init(sqlContext: SQLContext): Unit = {
        if (!initialized) {
          val cc = sqlContext.asInstanceOf[CarbonContext]
          val catalog = new CarbonMetastore(cc, cc.storePath, cc.hiveClientInterface, "")
          carbonEnv = CarbonEnv(catalog)
          CarbonIUDAnalysisRule.init(sqlContext)
          initialized = true
          CarbonProperties.getInstance.addProperty(CarbonCommonConstants.IS_DRIVER_INSTANCE, "true")
        }
      }
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user chenliang613 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/606
 
    retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---