[GitHub] carbondata pull request #2224: [CARBONDATA-2393]TaskNo is not working for SD...

[qiuchenjian-2]

GitHub user BJangir opened a pull request:

    https://github.com/apache/carbondata/pull/2224

    [CARBONDATA-2393]TaskNo is not working for SDK

    Issue:- Task No is not getting reflected in the Carbon Data file and in index file .
   
    Cause :- Task No is getting  overwritten  in CarbonTableOutputformate even CarbonModel have taskNo.
   
    Solution :- if CarbomModel has taskNo then no need to overwrite .
   
   
   
     - [ ] Any interfaces changed?
     NO
     - [ ] Any backward compatibility impacted?
     NO
     - [ ] Document update required?
    NO
     - [ ] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
    added TestCase
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
    NA


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/BJangir/incubator-carbondata CARBONDATA-2393

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2224.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2224
   
----
commit 6b19070ddcf632583b041fa7bee4111aaa3dc32e
Author: BJangir <babulaljangir111@...>
Date:   2018-04-24T14:42:54Z

    [CARBONDATA-2393] TaskNo is not working for SDK

----


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4513/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4205/



---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2224#discussion_r183800279
 
    --- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CSVCarbonWriterTest.java ---
    @@ -205,4 +205,70 @@ public void testSchemaPersistence() throws IOException {
         FileUtils.deleteDirectory(new File(path));
       }
     
    +  @Test
    +  public void testTaskNo() throws IOException {
    +    // TODO: write all data type and read by CarbonRecordReader to verify the content
    +    String path = "./testWriteFiles";
    +    FileUtils.deleteDirectory(new File(path));
    +
    +    Field[] fields = new Field[2];
    +    fields[0] = new Field("stringField", DataTypes.STRING);
    +    fields[1] = new Field("intField", DataTypes.INT);
    +
    +
    +    try {
    +      CarbonWriterBuilder builder = CarbonWriter.builder()
    +          .withSchema(new Schema(fields))
    +          .isTransactionalTable(true).taskNo("5")
    +          .outputPath(path);
    +
    +      CarbonWriter writer = builder.buildWriterForCSVInput();
    +
    +      for (int i = 0; i < 2; i++) {
    +        String[] row = new String[]{
    +            "robot" + (i % 10),
    +            String.valueOf(i)
    +        };
    +        writer.write(row);
    +      }
    +      writer.close();
    +    } catch (Exception e) {
    +      e.printStackTrace();
    +      Assert.fail(e.getMessage());
    +    }
    +
    +    File segmentFolder = new File(CarbonTablePath.getSegmentPath(path, "null"));
    +    Assert.assertTrue(segmentFolder.exists());
    +
    +    File[] dataFiles = segmentFolder.listFiles(new FileFilter() {
    +      @Override public boolean accept(File pathname) {
    +        return pathname.getName().endsWith(CarbonCommonConstants.FACT_FILE_EXT);
    +      }
    +    });
    +    Assert.assertNotNull(dataFiles);
    +    Assert.assertTrue(dataFiles.length > 0);
    +    String[] splitedcarbonFileName = getSplitedcarbonFileName(dataFiles[0]);
    --- End diff --
   
    Use CarbonTablePath.getTaskNo method


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2224#discussion_r183800623
 
    --- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CSVCarbonWriterTest.java ---
    @@ -205,4 +205,70 @@ public void testSchemaPersistence() throws IOException {
         FileUtils.deleteDirectory(new File(path));
       }
     
    +  @Test
    +  public void testTaskNo() throws IOException {
    +    // TODO: write all data type and read by CarbonRecordReader to verify the content
    +    String path = "./testWriteFiles";
    +    FileUtils.deleteDirectory(new File(path));
    +
    +    Field[] fields = new Field[2];
    +    fields[0] = new Field("stringField", DataTypes.STRING);
    +    fields[1] = new Field("intField", DataTypes.INT);
    +
    +
    +    try {
    +      CarbonWriterBuilder builder = CarbonWriter.builder()
    +          .withSchema(new Schema(fields))
    +          .isTransactionalTable(true).taskNo("5")
    +          .outputPath(path);
    +
    +      CarbonWriter writer = builder.buildWriterForCSVInput();
    +
    +      for (int i = 0; i < 2; i++) {
    +        String[] row = new String[]{
    +            "robot" + (i % 10),
    +            String.valueOf(i)
    +        };
    +        writer.write(row);
    +      }
    +      writer.close();
    +    } catch (Exception e) {
    +      e.printStackTrace();
    +      Assert.fail(e.getMessage());
    +    }
    +
    +    File segmentFolder = new File(CarbonTablePath.getSegmentPath(path, "null"));
    +    Assert.assertTrue(segmentFolder.exists());
    +
    +    File[] dataFiles = segmentFolder.listFiles(new FileFilter() {
    +      @Override public boolean accept(File pathname) {
    +        return pathname.getName().endsWith(CarbonCommonConstants.FACT_FILE_EXT);
    +      }
    +    });
    +    Assert.assertNotNull(dataFiles);
    +    Assert.assertTrue(dataFiles.length > 0);
    +    String[] splitedcarbonFileName = getSplitedcarbonFileName(dataFiles[0]);
    --- End diff --
   
    call this in finally block FileUtils.deleteDirectory(new File(path));


---

[qiuchenjian-2]

Github user BJangir commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2224#discussion_r183814030
 
    --- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CSVCarbonWriterTest.java ---
    @@ -205,4 +205,70 @@ public void testSchemaPersistence() throws IOException {
         FileUtils.deleteDirectory(new File(path));
       }
     
    +  @Test
    +  public void testTaskNo() throws IOException {
    +    // TODO: write all data type and read by CarbonRecordReader to verify the content
    +    String path = "./testWriteFiles";
    +    FileUtils.deleteDirectory(new File(path));
    +
    +    Field[] fields = new Field[2];
    +    fields[0] = new Field("stringField", DataTypes.STRING);
    +    fields[1] = new Field("intField", DataTypes.INT);
    +
    +
    +    try {
    +      CarbonWriterBuilder builder = CarbonWriter.builder()
    +          .withSchema(new Schema(fields))
    +          .isTransactionalTable(true).taskNo("5")
    +          .outputPath(path);
    +
    +      CarbonWriter writer = builder.buildWriterForCSVInput();
    +
    +      for (int i = 0; i < 2; i++) {
    +        String[] row = new String[]{
    +            "robot" + (i % 10),
    +            String.valueOf(i)
    +        };
    +        writer.write(row);
    +      }
    +      writer.close();
    +    } catch (Exception e) {
    +      e.printStackTrace();
    +      Assert.fail(e.getMessage());
    +    }
    +
    +    File segmentFolder = new File(CarbonTablePath.getSegmentPath(path, "null"));
    +    Assert.assertTrue(segmentFolder.exists());
    +
    +    File[] dataFiles = segmentFolder.listFiles(new FileFilter() {
    +      @Override public boolean accept(File pathname) {
    +        return pathname.getName().endsWith(CarbonCommonConstants.FACT_FILE_EXT);
    +      }
    +    });
    +    Assert.assertNotNull(dataFiles);
    +    Assert.assertTrue(dataFiles.length > 0);
    +    String[] splitedcarbonFileName = getSplitedcarbonFileName(dataFiles[0]);
    --- End diff --
   
    OK.Fixed


---

[qiuchenjian-2]

Github user BJangir commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    retest please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4516/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4208/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5375/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5529/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5532/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4367/



---

[qiuchenjian-2]

Github user BJangir commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4632/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5536/



---

[qiuchenjian-2]

Github user BJangir commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4373/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2224
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5539/



---