[GitHub] carbondata pull request #3021: Cdh support

[qiuchenjian-2]

GitHub user chandrasaripaka opened a pull request:

    https://github.com/apache/carbondata/pull/3021

    Cdh support

    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ No] Any interfaces changed?
     
     - [ No] Any backward compatibility impacted?
     
     - [ Yes] Document update required?
   
     - [ Yes] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ No Large Changes] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   
    This is just only the way we fix for CDH5.14.2, and the Spark2.2.0, the way how parquet treats the data.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/DataHeaps/carbondata cdh-support

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/3021.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3021
   
----
commit 746657e7e47fc69a62dbf14b3a7449108b92342c
Author: Chandrasekhar Saripaka <chandru.chanda@...>
Date:   2018-07-25T17:10:20Z

    Merge pull request #1 from apache/master
   
    merge back

commit 0c4c5f5d43c779c62864ed7ab99cf9ef91b082f2
Author: Chandrasekhar Saripaka <chandru.chanda@...>
Date:   2018-11-23T12:24:19Z

    Merge pull request #3 from apache/master
   
    Merge Carbondata master

commit 91d4ee54bfac7bad56dede27059896c766b5f42f
Author: Chandrasekhar Saripaka <chandru.chanda@...>
Date:   2018-12-13T07:42:30Z

    Merge pull request #4 from apache/master
   
    Updating the fork master

commit 834e702d0e2e8211987c465e36596d78028ea8b2
Author: Chandrasekhar Saripaka <chandru.chanda@...>
Date:   2018-12-14T13:36:26Z

    Merge pull request #5 from apache/master
   
    Merge Master to DataHeaps Master

commit 2ebfdf3142ee3f22e7938d735ca5a88c117f0848
Author: Chandrasekhar Saripaka <chandru.chanda@...>
Date:   2018-12-21T08:51:13Z

    Merge pull request #6 from apache/master
   
    Merge Upstream Master

commit 0c777259d54d3b99acd9d62b75cddde7e726afc1
Author: ChandraSaripaka <chandras4bigdata@...>
Date:   2018-12-24T07:19:30Z

    CARBONDATA-3193: Build support for CDH 5.14.2 Spark 2.2.0

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1923/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10176/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2132/



---

[qiuchenjian-2]

Github user chandrasaripaka commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    @jackylk , @ravipesala Can somebody validate this against the CDH5.14.2 and could you please create a profile for build validation for CDH Libraries.. Please let me know where to add..if I need to trigger a different profile build.


---

[qiuchenjian-2]

Github user qiuchenjian commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    Does carbon not support Cdh5.14.2 spark2.2.0  using -Pspark-2.2 ?
    Did CDH change the spark interface ,so that carbon can't run succesfully ?


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    @chandrasaripaka I can't find the spark maven dependency for CDH5.14.2, But I am able to build with CDH spark versions `2.2.0-cdh6.0.1` and `2.2.0.cloudera3`. Only the problem here I found is it does not have `spark-hive-thriftserver` jar in cloudera repo, so classes related it like `CarbonThriftServer` and `CarbonSQLCLIDriver` cannot compile. Apart from I am able to compile carbon with `2.2.0-cdh6.0.1` and `2.2.0.cloudera3` versions. I am not sure why CDH does not include `spark-hive-thriftserver` jar in there repo.
    Please send the repository for CDH5.14.2, so that I can verify this version also.



---

[qiuchenjian-2]

Github user chandrasaripaka commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    Please check the repo here.
    https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh5_maven_repo_514x.html#maven_5142


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    I went through this link earlier, but I cannot find spark 2.2 version in this distribution. I can find only `1.6.0-cdh5.14.4` of spark here.


---

[qiuchenjian-2]

Github user chandrasaripaka commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    Please try this:
    https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-sql_2.11/2.2.0.cloudera2/


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    @chandrasaripaka , I got the issue, but creating many duplicate files may not be a good idea as it will be difficult to maintain, I will try to do with reflection.
    And one more question is I don't find the package `spark-hive-thriftserver_2.11` from cloudera, without this package we cannot run the carbon thrift server. Where to find this package? Or Is it ok if we don't run carbon thrift server from cloudera distribution?


---

[qiuchenjian-2]

Github user chandrasaripaka commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    @ravipesala , Yes you can let me know how you want to inherit..I think defining a interface for the wrapper would be a good fit.. In our local fork we did it using spark-2.2.0 normal thrift server, it is ok if we dont run thrift server from cloudera distribution, that works.


---

[qiuchenjian-2]

Github user chenliang613 commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    @chandrasaripaka  
    As I know, spark 2.2.0 is not a stable version, it is better to consider other more stable versions.


---

[qiuchenjian-2]

Github user chandrasaripaka commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    @chenliang613 , I am ok with it..as in the corporate we still need to livw with it.. we can update the jira and close it.


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    @chandrasaripaka Please check the PR https://github.com/apache/carbondata/pull/3026


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1953/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10206/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2228/



---

[qiuchenjian-2]

Github user chandrasaripaka commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    We get error from a commit related to altertableschema.. which is not there in cloudera spark


---

[qiuchenjian-2]

Github user chandrasaripaka commented on the issue:

    https://github.com/apache/carbondata/pull/3021
 
    https://github.com/apache/carbondata/commit/b0733ecbf380d7956dee57a9048dd7537620744e


---