[GitHub] carbondata pull request #2583: [CARBONDATA-2803]fix wrong datasize calculati...

[qiuchenjian-2]

GitHub user akashrn5 opened a pull request:

    https://github.com/apache/carbondata/pull/2583

    [CARBONDATA-2803]fix wrong datasize calculation

    ### Problem: When a single block has multiple blocklets, then the data size written in table status file was wrong.
   
    ### Solution:
    The indexFileMap contains all the blocklets and index file mapping. For example, if one block contains 3 blocklets, then index file entry will list all the blocklets of all the block present in it. Since all the three blocklets will have the same block path, so just get the size of one block path for exact data size and avoid wrong datasize calculation.
   
    How this ### tested:
    tested with 1TB block, without fix it was giving three times size as data size, and after fix gives the correct data size.
   
   
    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [x] Any interfaces changed?
     NA
     - [x] Any backward compatibility impacted?
     NA
     - [x] Document update required?
    NA
     - [x] Testing done
    Done in 3 node cluster
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
    NA


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/akashrn5/incubator-carbondata datasize

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2583.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2583
   
----
commit 30ba2f50cf68f4f0af1278b26c0408243919997e
Author: akashrn5 <akashnilugal@...>
Date:   2018-07-30T14:11:34Z

    fix wrong datasize calculation

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7623/



---

[qiuchenjian-2]

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    retest this please



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6353/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7634/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6066/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6374/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    LGTM


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7655/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7661/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6390/



---

[qiuchenjian-2]

Github user manishgupta88 commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    LGTM


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6076/



---

[qiuchenjian-2]

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    retest sdv please



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2583#discussion_r207097923
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
    @@ -2651,8 +2651,17 @@ public static int isFilterPresent(byte[][] filterValues,
           carbonIndexSize = getCarbonIndexSize(fileStore, locationMap);
           for (Map.Entry<String, List<String>> entry : indexFilesMap.entrySet()) {
             // get the size of carbondata files
    +        String tempBlockFilePath = null;
             for (String blockFile : entry.getValue()) {
    -          carbonDataSize += FileFactory.getCarbonFile(blockFile).getSize();
    +          // the indexFileMap contains all the blocklets and index file mapping. For example, if one
    +          // block contains 3 blocklets, then entry.getValue() will list all the blocklets of all
    +          // the block present in it. Since all the three blocklets will have the same block path,
    +          // so just get the size of one block path for exact data size and avoid wrong datasize
    +          // calculation.
    +          if (!blockFile.equals(tempBlockFilePath)) {
    --- End diff --
   
    I feel this fix is not required, Please check PR https://github.com/apache/carbondata/pull/2596 to avoid duplicates from indexfileMap


---

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2583#discussion_r207103282
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
    @@ -2651,8 +2651,17 @@ public static int isFilterPresent(byte[][] filterValues,
           carbonIndexSize = getCarbonIndexSize(fileStore, locationMap);
           for (Map.Entry<String, List<String>> entry : indexFilesMap.entrySet()) {
             // get the size of carbondata files
    +        String tempBlockFilePath = null;
             for (String blockFile : entry.getValue()) {
    -          carbonDataSize += FileFactory.getCarbonFile(blockFile).getSize();
    +          // the indexFileMap contains all the blocklets and index file mapping. For example, if one
    +          // block contains 3 blocklets, then entry.getValue() will list all the blocklets of all
    +          // the block present in it. Since all the three blocklets will have the same block path,
    +          // so just get the size of one block path for exact data size and avoid wrong datasize
    +          // calculation.
    +          if (!blockFile.equals(tempBlockFilePath)) {
    --- End diff --
   
    ok, i will check and remove this change


---

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2583#discussion_r207113233
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
    @@ -2651,8 +2651,17 @@ public static int isFilterPresent(byte[][] filterValues,
           carbonIndexSize = getCarbonIndexSize(fileStore, locationMap);
           for (Map.Entry<String, List<String>> entry : indexFilesMap.entrySet()) {
             // get the size of carbondata files
    +        String tempBlockFilePath = null;
             for (String blockFile : entry.getValue()) {
    -          carbonDataSize += FileFactory.getCarbonFile(blockFile).getSize();
    +          // the indexFileMap contains all the blocklets and index file mapping. For example, if one
    +          // block contains 3 blocklets, then entry.getValue() will list all the blocklets of all
    +          // the block present in it. Since all the three blocklets will have the same block path,
    +          // so just get the size of one block path for exact data size and avoid wrong datasize
    +          // calculation.
    +          if (!blockFile.equals(tempBlockFilePath)) {
    --- End diff --
   
    i will revert this change


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7726/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6452/



---

[qiuchenjian-2]

Github user akashrn5 commented on the issue:

    https://github.com/apache/carbondata/pull/2583
 
    retest this please


---