[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2969#discussion_r242394285
 
    --- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
    @@ -31,7 +31,7 @@ object HiveExample {
     
       def main(args: Array[String]) {
    --- End diff --
   
    Please optimize this example and RunHiveExampleTest like:org.apache.carbondata.examplesCI.RunExamples.
     We should add the example to CI, and add some assert for exception and result. We should avoid that some developer/PR change code and lead to the hiveExample error,  the developer need fix it before their PR merged


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2969#discussion_r242395640
 
    --- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
    @@ -193,5 +196,4 @@ object HiveExample {
         }
         hiveEmbeddedServer2.stop()
    --- End diff --
   
    please fix the issue: the hiveExample don't stop after running the code. HiveExample should stop.


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2969#discussion_r242400169
 
    --- Diff: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java ---
    @@ -106,9 +107,17 @@ private static CarbonTable getCarbonTable(Configuration configuration, String pa
         CarbonInputSplit split;
         for (int i = 0; i < splitList.size(); i++) {
           split = (CarbonInputSplit) splitList.get(i);
    -      splits[i] = new CarbonHiveInputSplit(split.getSegmentId(), split.getPath(), split.getStart(),
    -          split.getLength(), split.getLocations(), split.getNumberOfBlocklets(), split.getVersion(),
    -          split.getBlockStorageIdMap());
    +      CarbonHiveInputSplit inputSplit = new CarbonHiveInputSplit(split.getSegmentId(),
    +              split.getPath(), split.getStart(), split.getLength(),
    +              split.getLocations(), split.getNumberOfBlocklets(),
    +              split.getVersion(), split.getBlockStorageIdMap());
    +      BlockletDetailInfo info = new BlockletDetailInfo();
    +      info.setBlockSize(split.getLength());
    +      info.setBlockFooterOffset(split.getDetailInfo().getBlockFooterOffset());
    +      info.setVersionNumber(split.getVersion().number());
    +      info.setUseMinMaxForPruning(false);
    --- End diff --
   
    Why do you set false in here?


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2969
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1835/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2969
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1836/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2969
 
    Build Failed  with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10093/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2969
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2044/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2969
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1837/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2969
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2045/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2969
 
    Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10094/



---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2969#discussion_r242814739
 
    --- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
    @@ -43,28 +43,25 @@ object HiveExample {
     
         import org.apache.spark.sql.CarbonSession._
     
    -    val carbonSession = SparkSession
    -      .builder()
    -      .master("local")
    -      .appName("HiveExample")
    -      .config("carbonSession.sql.warehouse.dir", warehouse).enableHiveSupport()
    -      .getOrCreateCarbonSession(
    -        store, metaStore_Db)
    +    val carbonSession = SparkSession.builder()
    +      .master("local").appName("HiveExample")
    --- End diff --
   
    Yes,I replace this with ExampleUtil to create carbon session.


---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2969#discussion_r242814939
 
    --- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
    @@ -43,28 +43,25 @@ object HiveExample {
     
         import org.apache.spark.sql.CarbonSession._
     
    -    val carbonSession = SparkSession
    -      .builder()
    -      .master("local")
    -      .appName("HiveExample")
    -      .config("carbonSession.sql.warehouse.dir", warehouse).enableHiveSupport()
    -      .getOrCreateCarbonSession(
    -        store, metaStore_Db)
    +    val carbonSession = SparkSession.builder()
    +      .master("local").appName("HiveExample")
    +      .enableHiveSupport()
    +      .config("spark.sql.warehouse.dir", warehouse)
    +      .getOrCreateCarbonSession(store, metaStore_Db)
     
         carbonSession.sql("""DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE""".stripMargin)
     
         carbonSession
           .sql(
    -        """CREATE TABLE HIVE_CARBON_EXAMPLE (ID int,NAME string,SALARY double) STORED BY
    -          |'CARBONDATA' """
    -          .stripMargin)
    +        "CREATE TABLE HIVE_CARBON_EXAMPLE (ID int,NAME string,SALARY double) " +
    +          "STORED BY 'CARBONDATA' ")
     
         carbonSession.sql(
           s"""
    -           LOAD DATA LOCAL INPATH '$rootPath/integration/hive/src/main/resources/data.csv' INTO
    -           TABLE
    -         HIVE_CARBON_EXAMPLE
    -           """)
    +               LOAD DATA LOCAL INPATH '$rootPath/integration/hive/src/main/resources/data.csv' INTO
    +               TABLE
    +             HIVE_CARBON_EXAMPLE
    --- End diff --
   
    Yeah, I didn't optimize this SQL format


---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2969#discussion_r242815043
 
    --- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
    @@ -85,19 +82,25 @@ object HiveExample {
     
         logger.info(s"============HIVE CLI IS STARTED ON PORT $port ==============")
     
    -    statement.execute("CREATE TABLE IF NOT EXISTS " + "HIVE_CARBON_EXAMPLE " +
    -                      " (ID int, NAME string,SALARY double)")
    +    statement.execute("DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE")
    +
    +    statement.execute("CREATE TABLE HIVE_CARBON_EXAMPLE " +
    +      " (ID int, NAME string,SALARY double) " +
    +      "ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe' " +
    +      "WITH SERDEPROPERTIES ('mapreduce.input.carboninputformat.databaseName'='default', " +
    +      "'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE')")
    +
         statement
           .execute(
             "ALTER TABLE HIVE_CARBON_EXAMPLE SET FILEFORMAT INPUTFORMAT \"org.apache.carbondata." +
    -        "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
    -        "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
    -        "CarbonHiveSerDe\" ")
    +          "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
    --- End diff --
   
    I have already added space before OUTPUTFORMAT


---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2969#discussion_r242815115
 
    --- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
    @@ -85,19 +82,25 @@ object HiveExample {
     
         logger.info(s"============HIVE CLI IS STARTED ON PORT $port ==============")
     
    -    statement.execute("CREATE TABLE IF NOT EXISTS " + "HIVE_CARBON_EXAMPLE " +
    -                      " (ID int, NAME string,SALARY double)")
    +    statement.execute("DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE")
    +
    +    statement.execute("CREATE TABLE HIVE_CARBON_EXAMPLE " +
    +      " (ID int, NAME string,SALARY double) " +
    +      "ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe' " +
    +      "WITH SERDEPROPERTIES ('mapreduce.input.carboninputformat.databaseName'='default', " +
    +      "'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE')")
    +
         statement
           .execute(
             "ALTER TABLE HIVE_CARBON_EXAMPLE SET FILEFORMAT INPUTFORMAT \"org.apache.carbondata." +
    -        "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
    -        "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
    -        "CarbonHiveSerDe\" ")
    +          "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
    +          "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
    --- End diff --
   
    I have already added space before SERDE


---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2969#discussion_r242815245
 
    --- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
    @@ -193,5 +196,4 @@ object HiveExample {
         }
         hiveEmbeddedServer2.stop()
    --- End diff --
   
    I have already solved hiveExample stop problem with system.exit()


---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2969#discussion_r242815339
 
    --- Diff: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java ---
    @@ -106,9 +107,17 @@ private static CarbonTable getCarbonTable(Configuration configuration, String pa
         CarbonInputSplit split;
         for (int i = 0; i < splitList.size(); i++) {
           split = (CarbonInputSplit) splitList.get(i);
    -      splits[i] = new CarbonHiveInputSplit(split.getSegmentId(), split.getPath(), split.getStart(),
    -          split.getLength(), split.getLocations(), split.getNumberOfBlocklets(), split.getVersion(),
    -          split.getBlockStorageIdMap());
    +      CarbonHiveInputSplit inputSplit = new CarbonHiveInputSplit(split.getSegmentId(),
    +              split.getPath(), split.getStart(), split.getLength(),
    +              split.getLocations(), split.getNumberOfBlocklets(),
    +              split.getVersion(), split.getBlockStorageIdMap());
    +      BlockletDetailInfo info = new BlockletDetailInfo();
    +      info.setBlockSize(split.getLength());
    +      info.setBlockFooterOffset(split.getDetailInfo().getBlockFooterOffset());
    +      info.setVersionNumber(split.getVersion().number());
    +      info.setUseMinMaxForPruning(false);
    --- End diff --
   
    There is no need to use min max for pruning.


---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2969#discussion_r242815829
 
    --- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
    @@ -31,7 +31,7 @@ object HiveExample {
     
       def main(args: Array[String]) {
    --- End diff --
   
    I updated RunHiveExampleTest and RunExamples as same as RunExamples.But this doesn't work in CI.I guess that the problem is caused by CI.


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2969#discussion_r242817509
 
    --- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
    @@ -85,19 +82,25 @@ object HiveExample {
     
         logger.info(s"============HIVE CLI IS STARTED ON PORT $port ==============")
     
    -    statement.execute("CREATE TABLE IF NOT EXISTS " + "HIVE_CARBON_EXAMPLE " +
    -                      " (ID int, NAME string,SALARY double)")
    +    statement.execute("DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE")
    +
    +    statement.execute("CREATE TABLE HIVE_CARBON_EXAMPLE " +
    +      " (ID int, NAME string,SALARY double) " +
    +      "ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe' " +
    +      "WITH SERDEPROPERTIES ('mapreduce.input.carboninputformat.databaseName'='default', " +
    +      "'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE')")
    +
         statement
           .execute(
             "ALTER TABLE HIVE_CARBON_EXAMPLE SET FILEFORMAT INPUTFORMAT \"org.apache.carbondata." +
    -        "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
    -        "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
    -        "CarbonHiveSerDe\" ")
    +          "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2969#discussion_r242817598
 
    --- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
    @@ -43,28 +43,25 @@ object HiveExample {
     
         import org.apache.spark.sql.CarbonSession._
     
    -    val carbonSession = SparkSession
    -      .builder()
    -      .master("local")
    -      .appName("HiveExample")
    -      .config("carbonSession.sql.warehouse.dir", warehouse).enableHiveSupport()
    -      .getOrCreateCarbonSession(
    -        store, metaStore_Db)
    +    val carbonSession = SparkSession.builder()
    +      .master("local").appName("HiveExample")
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2969
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1852/



---