[GitHub] carbondata pull request #2886: [WIP]make inverted index false by defaut

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1241/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1457/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9506/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1245/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1461/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9510/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1247/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1463/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9512/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2886#discussion_r230653013
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/dataload/TestNoInvertedIndexLoadAndQuery.scala ---
    @@ -289,6 +292,44 @@ class TestNoInvertedIndexLoadAndQuery extends QueryTest with BeforeAndAfterAll {
         checkAnswer(sql("""select c2 from testNull where c2 is null"""), Seq(Row(null), Row(null), Row(null), Row(null), Row(null), Row(null)))
       }
     
    +  test("inverted index with Dictionary_EXCLUDE and INVERTED_INDEX") {
    +    sql("drop table if exists index1")
    +    sql(
    +      """
    +           CREATE TABLE IF NOT EXISTS index1
    +           (id Int, name String, city String)
    +           STORED BY 'org.apache.carbondata.format'
    +           TBLPROPERTIES('DICTIONARY_EXCLUDE'='city','INVERTED_INDEX'='city')
    +      """)
    +    sql(
    +      s"""
    +           LOAD DATA LOCAL INPATH '$testData1' into table index1
    +           """)
    +    checkAnswer(
    +      sql(
    +        """
    +           SELECT * FROM index1 WHERE city = "Bangalore"
    +        """),
    +      Seq(Row(19.0, "Emily", "Bangalore")))
    --- End diff --
   
    Try to get the CarbonTable using CarbonMetadata.getInstance and verify the inverted index columns for all the testcases


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    @akashrn5 In existing flow inverted index is applied by default for all sort columns . No sort columns and measure columns cannot be applied for an inverted index. Is it the same case now also?  Are we allowing inverted index only for sort columns or allowing for all columns?  If allowed only for sort columns then please add the validations in parser and also in sdk.


---

[qiuchenjian-2]

Github user akashrn5 commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    @ravipesala
    in existing flow, all the sort columns and dimensions will have inverted index and,
     in current implementation, if measure columns are given in inverted_index property, then i ll not set it as inverted index. Basically i wont throw any exception. Do we need to throw exception?
    and currently im allowing the inverted index for all dimension columns and not restricting to sort columns only.
    So basically, user can mention columns in inverted_index columns (dimensions), i am setting those as inverted index. it is correct right


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    ok, its fine


---

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2886#discussion_r230684559
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/dataload/TestNoInvertedIndexLoadAndQuery.scala ---
    @@ -289,6 +292,44 @@ class TestNoInvertedIndexLoadAndQuery extends QueryTest with BeforeAndAfterAll {
         checkAnswer(sql("""select c2 from testNull where c2 is null"""), Seq(Row(null), Row(null), Row(null), Row(null), Row(null), Row(null)))
       }
     
    +  test("inverted index with Dictionary_EXCLUDE and INVERTED_INDEX") {
    +    sql("drop table if exists index1")
    +    sql(
    +      """
    +           CREATE TABLE IF NOT EXISTS index1
    +           (id Int, name String, city String)
    +           STORED BY 'org.apache.carbondata.format'
    +           TBLPROPERTIES('DICTIONARY_EXCLUDE'='city','INVERTED_INDEX'='city')
    +      """)
    +    sql(
    +      s"""
    +           LOAD DATA LOCAL INPATH '$testData1' into table index1
    +           """)
    +    checkAnswer(
    +      sql(
    +        """
    +           SELECT * FROM index1 WHERE city = "Bangalore"
    +        """),
    +      Seq(Row(19.0, "Emily", "Bangalore")))
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1269/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1484/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9533/



---

[qiuchenjian-2]

Github user akashrn5 commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    2.2.1 and 2.3.1 failed test cases, which are not related and these are random


---

[qiuchenjian-2]

Github user akashrn5 commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2886
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1275/



---