[GitHub] carbondata pull request #2432: [CARBONDATA-2675][32K] Support config long_st...

[qiuchenjian-2]

GitHub user kevinjmh opened a pull request:

    https://github.com/apache/carbondata/pull/2432

    [CARBONDATA-2675][32K] Support config long_string_columns when create datamap

    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
   
     - [ ] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   
   
    Create datamap use select statement, but long string column is defined with StringType in the result dataframe if this column is selected. This PR allows to set long_string_columns property in dmproperties.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kevinjmh/carbondata longstr_datamap

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2432.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2432
   
----
commit 5b7b7bfcce8d33015f6ef47e2723918198b176b3
Author: Manhua <kevinjmh@...>
Date:   2018-06-30T02:53:41Z

    support config long_string_columns when create datamap

----


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5538/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5510/



---

[qiuchenjian-2]

Github user kevinjmh commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6684/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5541/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5514/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5543/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6689/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5517/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2432#discussion_r200019284
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateTableHelper.scala ---
    @@ -106,6 +106,24 @@ case class PreAggregateTableHelper(
         tableProperties.put(CarbonCommonConstants.FLAT_FOLDER,
           parentTable.getTableInfo.getFactTable.getTableProperties.asScala.getOrElse(
             CarbonCommonConstants.FLAT_FOLDER, CarbonCommonConstants.DEFAULT_FLAT_FOLDER))
    +
    +    // update column name with prefix for long_string_column
    +    val longStringColumn = tableProperties.get(CarbonCommonConstants.LONG_STRING_COLUMNS)
    +    if (longStringColumn != None) {
    +      val fieldNames = fields.map(_.column)
    +      val newLongStringColumn = longStringColumn.get.split(",").map(_.trim).map(colName => {
    --- End diff --
   
    change format to
    ```
    .map { colName =>
       xxx
    }
    ```


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2432#discussion_r200019591
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateTableHelper.scala ---
    @@ -106,6 +106,24 @@ case class PreAggregateTableHelper(
         tableProperties.put(CarbonCommonConstants.FLAT_FOLDER,
           parentTable.getTableInfo.getFactTable.getTableProperties.asScala.getOrElse(
             CarbonCommonConstants.FLAT_FOLDER, CarbonCommonConstants.DEFAULT_FLAT_FOLDER))
    +
    +    // update column name with prefix for long_string_column
    --- End diff --
   
    Can you describe what is the prefix, and why it is needed in the comment


---

[qiuchenjian-2]

Github user kevinjmh commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6763/



---

[qiuchenjian-2]

Github user kevinjmh commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    retest this please


---

[qiuchenjian-2]

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6789/



---

[qiuchenjian-2]

Github user kevinjmh commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    Is something wrong with the build environment?  Errors like below appear in the CI result link:
   
    > FileNotFoundException means that the credentials Jenkins is using is probably wrong. Or the user account does not have write access to the repo.
   
    > stderr: fatal: unable to access 'https://github.com/apache/carbondata.git/': Failed to connect to github.com port 443: Connection timed out
   



---

[qiuchenjian-2]

Github user kevinjmh commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    retest this please


---

[qiuchenjian-2]

Github user kevinjmh commented on the issue:

    https://github.com/apache/carbondata/pull/2432
 
    retest this please


---