[GitHub] carbondata pull request #3012: [CARBONDATA-3127]Fix the HiveExample & TestCa...

[qiuchenjian-2]

GitHub user SteNicholas opened a pull request:

    https://github.com/apache/carbondata/pull/3012

    [CARBONDATA-3127]Fix the HiveExample & TestCarbonSerde exception

    [CARBONDATA-3127]Hive module test case has been commented off,can' t run.
    This pull request fix TestCarbonSerde test case exception with maven compile.Method called deserializeAndSerializeLazySimple of TestCarbonSerde class calls the interface called serializeStartKey which is not exsit in CarbonHiveSerDe class.Just modify replace serializeStartKey with serialize and remove comments of TestCarbonSerde.
    Previous hive-integration HiveExample exists obvious bug that there is no data when select HIVE_CARBON_EXAMPLE table.I have already fixed the bug above in this pull request.Location was configured wrong after creating hive table.And block lack of detail info and footer offset when do querying hive table operation.Therefore I set detail info of block with footer offset through input split.
     - [x] Any interfaces changed?
            No
     - [x] Any backward compatibility impacted?
           No
     - [x] Document update required?
           No
     - [x] Testing done
           Modify TestCarbonSerde test
     - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
           No

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/SteNicholas/carbondata CARBONDATA-3127

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/3012.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3012
   
----
commit f403455748aa98a224559dac7e8953725d98b41c
Author: Nicholas Jiang <programgeek@...>
Date:   2018-12-20T20:14:09Z

    Hive Integration

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3012
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1889/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3012
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2098/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3012
 
    Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10143/



---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/3012#discussion_r243470291
 
    --- Diff: examples/spark2/pom.xml ---
    @@ -78,7 +83,12 @@
         <dependency>
           <groupId>org.apache.httpcomponents</groupId>
           <artifactId>httpclient</artifactId>
    -      <version>${httpclient.version}</version>
    +      <version>4.3.4</version>
    --- End diff --
   
    Please use the unify version


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/3012#discussion_r243470296
 
    --- Diff: examples/spark2/pom.xml ---
    @@ -78,7 +83,12 @@
         <dependency>
           <groupId>org.apache.httpcomponents</groupId>
           <artifactId>httpclient</artifactId>
    -      <version>${httpclient.version}</version>
    +      <version>4.3.4</version>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.apache.httpcomponents</groupId>
    +      <artifactId>httpcore</artifactId>
    +      <version>4.3-alpha1</version>
    --- End diff --
   
    Please use the unify version


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/3012#discussion_r243470427
 
    --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/HiveExample.scala ---
    @@ -30,44 +33,38 @@ object HiveExample {
       private val driverName: String = "org.apache.hive.jdbc.HiveDriver"
     
       def main(args: Array[String]) {
    -    val rootPath = new File(this.getClass.getResource("/").getPath
    -                            + "../../../..").getCanonicalPath
    -    val store = s"$rootPath/integration/hive/target/store"
    -    val warehouse = s"$rootPath/integration/hive/target/warehouse"
    -    val metaStore_Db = s"$rootPath/integration/hive/target/carbon_metaStore_db"
    -    val logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
    -    var resultId = ""
    -    var resultName = ""
    -    var resultSalary = ""
    -
    +    val carbonSession = ExampleUtils.createCarbonSession("HiveExample")
    +    exampleBody(carbonSession, CarbonProperties.getStorePath
    +      + CarbonCommonConstants.FILE_SEPARATOR
    +      + CarbonCommonConstants.DATABASE_DEFAULT_NAME)
    +    carbonSession.close()
     
    -    import org.apache.spark.sql.CarbonSession._
    +    System.exit(0)
    +  }
     
    -    val carbonSession = SparkSession
    -      .builder()
    -      .master("local")
    -      .appName("HiveExample")
    -      .config("carbonSession.sql.warehouse.dir", warehouse).enableHiveSupport()
    -      .getOrCreateCarbonSession(
    -        store, metaStore_Db)
    +  def exampleBody(sparkSession: SparkSession, store: String): Unit = {
    --- End diff --
   
    carbonSession is better


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/3012#discussion_r243470594
 
    --- Diff: examples/spark2/src/test/scala/org/apache/carbondata/examplesCI/RunExamples.scala ---
    @@ -19,12 +19,12 @@ package org.apache.carbondata.examplesCI
     
     import org.apache.spark.sql.test.util.QueryTest
     import org.scalatest.BeforeAndAfterAll
    -
     import org.apache.carbondata.examples._
     import org.apache.carbondata.core.constants.CarbonCommonConstants
     import org.apache.carbondata.core.util.CarbonProperties
     import org.apache.carbondata.examples.sdk.CarbonReaderExample
     import org.apache.carbondata.examples.sql.JavaCarbonSessionExample
    +import org.apache.spark.sql.test.TestQueryExecutor
    --- End diff --
   
    please move o another group


---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/3012#discussion_r243478043
 
    --- Diff: examples/spark2/pom.xml ---
    @@ -78,7 +83,12 @@
         <dependency>
           <groupId>org.apache.httpcomponents</groupId>
           <artifactId>httpclient</artifactId>
    -      <version>${httpclient.version}</version>
    +      <version>4.3.4</version>
    --- End diff --
   
    @xubo245 I have already use the unify version of http client library.


---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/3012#discussion_r243478363
 
    --- Diff: examples/spark2/pom.xml ---
    @@ -78,7 +83,12 @@
         <dependency>
           <groupId>org.apache.httpcomponents</groupId>
           <artifactId>httpclient</artifactId>
    -      <version>${httpclient.version}</version>
    +      <version>4.3.4</version>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.apache.httpcomponents</groupId>
    +      <artifactId>httpcore</artifactId>
    +      <version>4.3-alpha1</version>
    --- End diff --
   
    @xubo245 I have already use the unify version of http core library.


---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/3012#discussion_r243478424
 
    --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/HiveExample.scala ---
    @@ -30,44 +33,38 @@ object HiveExample {
       private val driverName: String = "org.apache.hive.jdbc.HiveDriver"
     
       def main(args: Array[String]) {
    -    val rootPath = new File(this.getClass.getResource("/").getPath
    -                            + "../../../..").getCanonicalPath
    -    val store = s"$rootPath/integration/hive/target/store"
    -    val warehouse = s"$rootPath/integration/hive/target/warehouse"
    -    val metaStore_Db = s"$rootPath/integration/hive/target/carbon_metaStore_db"
    -    val logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
    -    var resultId = ""
    -    var resultName = ""
    -    var resultSalary = ""
    -
    +    val carbonSession = ExampleUtils.createCarbonSession("HiveExample")
    +    exampleBody(carbonSession, CarbonProperties.getStorePath
    +      + CarbonCommonConstants.FILE_SEPARATOR
    +      + CarbonCommonConstants.DATABASE_DEFAULT_NAME)
    +    carbonSession.close()
     
    -    import org.apache.spark.sql.CarbonSession._
    +    System.exit(0)
    +  }
     
    -    val carbonSession = SparkSession
    -      .builder()
    -      .master("local")
    -      .appName("HiveExample")
    -      .config("carbonSession.sql.warehouse.dir", warehouse).enableHiveSupport()
    -      .getOrCreateCarbonSession(
    -        store, metaStore_Db)
    +  def exampleBody(sparkSession: SparkSession, store: String): Unit = {
    --- End diff --
   
    @xubo245 I update parameter name in exampleBody method according to your  suggestion.


---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/3012#discussion_r243478610
 
    --- Diff: examples/spark2/src/test/scala/org/apache/carbondata/examplesCI/RunExamples.scala ---
    @@ -19,12 +19,12 @@ package org.apache.carbondata.examplesCI
     
     import org.apache.spark.sql.test.util.QueryTest
     import org.scalatest.BeforeAndAfterAll
    -
     import org.apache.carbondata.examples._
     import org.apache.carbondata.core.constants.CarbonCommonConstants
     import org.apache.carbondata.core.util.CarbonProperties
     import org.apache.carbondata.examples.sdk.CarbonReaderExample
     import org.apache.carbondata.examples.sql.JavaCarbonSessionExample
    +import org.apache.spark.sql.test.TestQueryExecutor
    --- End diff --
   
    @xubo245 Yeah, I move to another group.TestCarbonSerDe is also updated as same as the way of RunExamples.scala.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3012
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1892/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3012
 
    Build Failed  with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10147/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3012
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2101/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3012
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1893/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3012
 
    Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10148/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3012
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2102/



---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/3012
 
    LGTM


---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/3012
 
    LGTM


---