[GitHub] carbondata pull request #1471: [WIP] Datamap FineGrain implementation

[qiuchenjian-2]

GitHub user ravipesala opened a pull request:

    https://github.com/apache/carbondata/pull/1471

    [WIP] Datamap FineGrain implementation

    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     Implemented interfaces for FG datamap and integrated to filterscanner to use the pruned bitset from FG datamap.
    FG Query flow as follows.
     1.The user can add FG datamap to any table and implement there interfaces.
     2. Any filter query which hits the table with datamap will call prune method of FGdatamap.
     3. The prune method of FGDatamap return list FineGrainBlocklet , these blocklets contain the information of block, blocklet, page and rowids information as well.
     4. The pruned  blocklets are internally wriitten to file and returns only the block , blocklet and filepath  information as part of Splits.
    5. Based on the splits scanrdd schedule the tasks.
    6. In filterscanner we check the datamapwriterpath from split and reNoteads the bitset if exists. And pass this bitset as input to it.
   
    NOTE : This PR depends on https://github.com/apache/carbondata/pull/1410
   
     - [X] Any interfaces changed?
           Few changes to datamap interfaces only.
     
     - [X] Any backward compatibility impacted? NO
     
     - [X] Document update required?
         Yes, Required to add to dev guide about interfaces
   
     - [X] Testing done
            Tests are added for both CG and FG
           
     - [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ravipesala/incubator-carbondata datamap-fg-impl

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1471.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1471
   
----
commit 8d727ffbc465191040274318a55dd0a69063bdfd
Author: ravipesala <[hidden email]>
Date:   2017-10-11T13:37:22Z

    Added FG interfaces

commit 0cc36e4b28d2747728013116cdacecb80beb0239
Author: ravipesala <[hidden email]>
Date:   2017-10-17T12:23:09Z

    Refactored code to remove path from blocklet

commit 3c204c2d2e69512f21f630a091df756255ccd6e8
Author: ravipesala <[hidden email]>
Date:   2017-10-31T09:55:57Z

    fgdatamap implement

commit 717ba3e41a6f3d1936ff14857a9b992cff13c8a6
Author: ravipesala <[hidden email]>
Date:   2017-11-03T16:52:33Z

    Added test for FGDatamap

commit e6093540c800c2e86c472787603c884a72e1d129
Author: ravipesala <[hidden email]>
Date:   2017-11-04T06:21:21Z

    Added testcase

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/833/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1466/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1468/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/835/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1474/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/841/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/968/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1582/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1087/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1090/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1688/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1691/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1100/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1700/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1107/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1706/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1132/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1135/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1471
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1723/



---