[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2190
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5257/



---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2190#discussion_r183225423
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/executor/util/RestructureUtil.java ---
    @@ -355,12 +338,11 @@ public static Object getMeasureDefaultValueByType(ColumnSchema columnSchema,
        * @param blockExecutionInfo
        * @param queryMeasures        measures present in query
        * @param currentBlockMeasures current block measures
    -   * @param isUnManagedTable
        * @return measures present in the block
        */
       public static List<ProjectionMeasure> createMeasureInfoAndGetCurrentBlockQueryMeasures(
           BlockExecutionInfo blockExecutionInfo, List<ProjectionMeasure> queryMeasures,
    -      List<CarbonMeasure> currentBlockMeasures, boolean isUnManagedTable) {
    +      List<CarbonMeasure> currentBlockMeasures) {
    --- End diff --
   
    For non transactional table columnid check should not be persent during read. As files can be created some where else and can be copied.


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2190#discussion_r183225429
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java ---
    @@ -196,11 +235,26 @@ private CarbonTable buildCarbonTable() {
           tableSchemaBuilder = tableSchemaBuilder.blockletSize(blockletSize);
         }
     
    -    List<String> sortColumnsList;
    -    if (sortColumns != null) {
    -      sortColumnsList = Arrays.asList(sortColumns);
    +    if (isUnManagedTable) {
    +      tableSchemaBuilder = tableSchemaBuilder.isUnmanagedTable(isUnManagedTable);
    +    }
    +
    +    List<String> sortColumnsList = new ArrayList<>();
    --- End diff --
   
    Sort options behaviour should be consistent as create table
    If empty sort columns list, then it should be no sort.
    If sort column list not passed, default behaviour should be same as create table.


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2190#discussion_r183225457
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java ---
    @@ -119,7 +126,18 @@ public TableSchemaBuilder addColumn(StructField field, boolean isSortColumn) {
         }
         newColumn.setSchemaOrdinal(ordinal++);
         newColumn.setColumnar(true);
    -    newColumn.setColumnUniqueId(UUID.randomUUID().toString());
    +
    +    // For unmanagedTable, multiple sdk writer output with same column name can be placed in
    --- End diff --
   
    Change un managed to transactional table in all PR specific code added.


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2190#discussion_r183225463
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java ---
    @@ -123,6 +125,43 @@ public CarbonWriterBuilder uniqueIdentifier(long UUID) {
         return this;
       }
     
    +  /**
    +   * To support the load options for sdk writer
    +   * @param options key,value pair of load options.
    +   *                supported keys values are
    +   *                a. bad_records_logger_enable -- true, false
    +   *                b. bad_records_action -- FAIL, FORCE, IGNORE, REDIRECT
    +   *                c. bad_record_path -- path
    +   *                d. dateformat -- same as JAVA SimpleDateFormat
    +   *                e. timestampformat -- same as JAVA SimpleDateFormat
    +   * @return updated CarbonWriterBuilder
    +   */
    +  public CarbonWriterBuilder withLoadOptions(Map<String, String> options) {
    --- End diff --
   
    complex type level delimiters should be supported, quote char, escape character needs to be supported, as required for complex types parsing.


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2190#discussion_r183225467
 
    --- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
    @@ -48,11 +48,11 @@ public void testWriteAndReadFiles() throws IOException, InterruptedException {
     
         int i = 0;
         while (reader.hasNext()) {
    -      Object[] row = (Object[])reader.readNextRow();
    -      Assert.assertEquals("robot" + (i % 10), row[0]);
    -      Assert.assertEquals(i, row[1]);
    +      Object[] row = (Object[]) reader.readNextRow();
    +      // TODO: Default sort column is applied for dimensions. So, need  to validate accordingly
    --- End diff --
   
    TODO: to be corrected


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2190#discussion_r183225480
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java ---
    @@ -123,6 +125,43 @@ public CarbonWriterBuilder uniqueIdentifier(long UUID) {
         return this;
       }
     
    +  /**
    +   * To support the load options for sdk writer
    +   * @param options key,value pair of load options.
    +   *                supported keys values are
    +   *                a. bad_records_logger_enable -- true, false
    --- End diff --
   
     please follow standard documentation format


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2190
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5296/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2190
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4117/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2190#discussion_r183227700
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java ---
    @@ -119,7 +126,18 @@ public TableSchemaBuilder addColumn(StructField field, boolean isSortColumn) {
         }
         newColumn.setSchemaOrdinal(ordinal++);
         newColumn.setColumnar(true);
    -    newColumn.setColumnUniqueId(UUID.randomUUID().toString());
    +
    +    // For unmanagedTable, multiple sdk writer output with same column name can be placed in
    --- End diff --
   
    ok. done


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2190#discussion_r183227711
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java ---
    @@ -123,6 +125,43 @@ public CarbonWriterBuilder uniqueIdentifier(long UUID) {
         return this;
       }
     
    +  /**
    +   * To support the load options for sdk writer
    +   * @param options key,value pair of load options.
    +   *                supported keys values are
    +   *                a. bad_records_logger_enable -- true, false
    +   *                b. bad_records_action -- FAIL, FORCE, IGNORE, REDIRECT
    +   *                c. bad_record_path -- path
    +   *                d. dateformat -- same as JAVA SimpleDateFormat
    +   *                e. timestampformat -- same as JAVA SimpleDateFormat
    +   * @return updated CarbonWriterBuilder
    +   */
    +  public CarbonWriterBuilder withLoadOptions(Map<String, String> options) {
    --- End diff --
   
    ok. done


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2190#discussion_r183227722
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java ---
    @@ -196,11 +235,26 @@ private CarbonTable buildCarbonTable() {
           tableSchemaBuilder = tableSchemaBuilder.blockletSize(blockletSize);
         }
     
    -    List<String> sortColumnsList;
    -    if (sortColumns != null) {
    -      sortColumnsList = Arrays.asList(sortColumns);
    +    if (isUnManagedTable) {
    +      tableSchemaBuilder = tableSchemaBuilder.isUnmanagedTable(isUnManagedTable);
    +    }
    +
    +    List<String> sortColumnsList = new ArrayList<>();
    --- End diff --
   
    ok. done


---

[qiuchenjian-2]

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2190
 
    Retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2190
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4119/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2190
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5299/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2190
 
    Retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2190
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4121/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2190
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5301/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2190
 
    Retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2190
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4123/



---