[GitHub] carbondata pull request #2420: [WIP] updated rename command so that table di...

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5572/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5557/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6727/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6730/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5576/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5564/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6732/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5578/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5565/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5579/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6734/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6740/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5587/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5574/



---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2420#discussion_r200150569
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java ---
    @@ -345,14 +346,18 @@ public static CarbonLoadModel getLoadModel(Configuration conf) throws IOExceptio
                         CarbonCommonConstants.LOAD_BATCH_SORT_SIZE_INMB,
                         CarbonCommonConstants.LOAD_BATCH_SORT_SIZE_INMB_DEFAULT))));
     
    -    model.setBadRecordsLocation(
    -        conf.get(BAD_RECORD_PATH,
    -            carbonProperty.getProperty(
    -                CarbonLoadOptionConstants.CARBON_OPTIONS_BAD_RECORD_PATH,
    -                carbonProperty.getProperty(
    -                    CarbonCommonConstants.CARBON_BADRECORDS_LOC,
    -                    CarbonCommonConstants.CARBON_BADRECORDS_LOC_DEFAULT_VAL))));
    -
    +    String badRecordsPath = conf.get(BAD_RECORD_PATH);
    +    if (StringUtils.isEmpty(badRecordsPath)) {
    +      badRecordsPath =
    --- End diff --
   
    carbon table properties should be checked first then take from configuration


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2420#discussion_r200151855
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/streaming/StreamSinkFactory.scala ---
    @@ -255,6 +256,29 @@ object StreamSinkFactory {
           optionsFinal.put("fileheader", carbonTable.getCreateOrderColumn(carbonTable.getTableName)
             .asScala.map(_.getColName).mkString(","))
         }
    +    val badRecordsFromLoad = parameters.get("bad_record_path")
    +    val badRecordsFromCreate = carbonTable.getTableInfo.getFactTable.getTableProperties
    +      .get("bad_records_path")
    +    val badRecordsPath = if (badRecordsFromLoad.isDefined &&
    --- End diff --
   
    badRecordsFromLoad.isDefined  check not required


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2420#discussion_r200155277
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/model/CarbonLoadModelBuilder.java ---
    @@ -72,6 +72,28 @@ public CarbonLoadModel build(Map<String, String> options, long UUID, String task
           }
           optionsFinal.put("fileheader", Strings.mkString(columns, ","));
         }
    +    String badRecordsFromLoad = options.get("bad_record_path");
    +    String badRecordsFromCreate =
    +        table.getTableInfo().getFactTable().getTableProperties().get("bad_records_path");
    --- End diff --
   
    Move this logic to common place, related to badrecords, this logic repeated so many times


---

[qiuchenjian-2]

Github user kunal642 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2420#discussion_r200166867
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java ---
    @@ -345,14 +346,18 @@ public static CarbonLoadModel getLoadModel(Configuration conf) throws IOExceptio
                         CarbonCommonConstants.LOAD_BATCH_SORT_SIZE_INMB,
                         CarbonCommonConstants.LOAD_BATCH_SORT_SIZE_INMB_DEFAULT))));
     
    -    model.setBadRecordsLocation(
    -        conf.get(BAD_RECORD_PATH,
    -            carbonProperty.getProperty(
    -                CarbonLoadOptionConstants.CARBON_OPTIONS_BAD_RECORD_PATH,
    -                carbonProperty.getProperty(
    -                    CarbonCommonConstants.CARBON_BADRECORDS_LOC,
    -                    CarbonCommonConstants.CARBON_BADRECORDS_LOC_DEFAULT_VAL))));
    -
    +    String badRecordsPath = conf.get(BAD_RECORD_PATH);
    +    if (StringUtils.isEmpty(badRecordsPath)) {
    +      badRecordsPath =
    --- End diff --
   
    conf.get will give bad_record_path from load options


---

[qiuchenjian-2]

Github user kunal642 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2420#discussion_r200167194
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/streaming/StreamSinkFactory.scala ---
    @@ -255,6 +256,29 @@ object StreamSinkFactory {
           optionsFinal.put("fileheader", carbonTable.getCreateOrderColumn(carbonTable.getTableName)
             .asScala.map(_.getColName).mkString(","))
         }
    +    val badRecordsFromLoad = parameters.get("bad_record_path")
    +    val badRecordsFromCreate = carbonTable.getTableInfo.getFactTable.getTableProperties
    +      .get("bad_records_path")
    +    val badRecordsPath = if (badRecordsFromLoad.isDefined &&
    --- End diff --
   
    badRecordsFromLoad if Option[String] to get the value from it i need to check isDefined and then get the value.


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2420
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5612/



---