[GitHub] carbondata pull request #1735: [CARBONDATA-1947]fix select * issue after com...

[qiuchenjian-2]

GitHub user akashrn5 opened a pull request:

    https://github.com/apache/carbondata/pull/1735

    [CARBONDATA-1947]fix select * issue after compaction, delete and clean files operation

    **Problem**: fix select * issue after compaction, delete and clean files operation
   
    **Analysis**: during compaction the timestamp of fact file and the load starttime present in metadata details are different for the same segments, so when the delete operation is executed and after that clean files is called,  since there is a timestamp difference, it will delete the carbondata file which inturn gives empty result for select query.
   
    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [X] Any interfaces changed?
     NA
     - [X] Any backward compatibility impacted?
     NA
     - [X] Document update required?
    NA
     - [X] Testing done
            Please provide details on
            - Units test cases are added to test the scenario
           
     - [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
    NA


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/akashrn5/incubator-carbondata clean_after_delete

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1735.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1735
   
----
commit 47e9a7753473936607f54321d7b0602fcf50e90a
Author: akashrn5 <akashnilugal@...>
Date:   2017-12-28T08:13:15Z

    fix select * issue after compaction, delete and clean files operation

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1735
 
    Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1179/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1735
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2402/



---

[qiuchenjian-2]

Github user akashrn5 commented on the issue:

    https://github.com/apache/carbondata/pull/1735
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1735
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2405/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1735
 
    Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1180/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1735#discussion_r158956869
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala ---
    @@ -165,6 +165,20 @@ class DeleteCarbonTableTestCase extends QueryTest with BeforeAndAfterAll {
         sql("drop table if exists preaggmain_preagg1")
       }
     
    +  test("test select query after compaction, delete and clean files") {
    +    sql("drop table if exists select_after_clean")
    +    sql("create table select_after_clean(id int, name string) stored by 'carbondata'")
    +    sql("insert into select_after_clean select 1,'abc'")
    +    sql("insert into select_after_clean select 2,'def'")
    +    sql("insert into select_after_clean select 3,'uhj'")
    +    sql("insert into select_after_clean select 4,'frg'")
    +    sql("alter table select_after_clean compact 'minor'")
    +    sql("clean files for table select_after_clean")
    +    sql("delete from select_after_clean where name='def'")
    +    sql("clean files for table select_after_clean")
    +    assert(sql("select * from select_after_clean").collect().length == 3)
    --- End diff --
   
    can you assert the result using checkAnswer


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1735
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2419/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1735
 
    Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1195/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1735
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2602/



---

[qiuchenjian-2]

Github user akashrn5 commented on the issue:

    https://github.com/apache/carbondata/pull/1735
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1735
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2611/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1735
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2427/



---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/1735
 
    LGTM


---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/1735


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1735
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1203/



---