[GitHub] carbondata pull request #2281: [CARBONDATA-2455]To support TEXT_MATCH to sea...

[qiuchenjian-2]

GitHub user Indhumathi27 opened a pull request:

    https://github.com/apache/carbondata/pull/2281

    [CARBONDATA-2455]To support TEXT_MATCH to search empty string with Wildcard(*) and AND,OR,NOT Filter fix

   
     - [x] Any interfaces changed?
             NA
     - [x] Any backward compatibility impacted?
             NA
     - [x] Document update required?
            YES
     - [x] Testing done
            Yes. Testcases added
     - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
            NA


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Indhumathi27/carbondata luceneempty

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2281.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2281
   
----
commit ebee9236c0226d9829796818f731f2c9aefcc7ae
Author: Indhumathi27 <indhumathim27@...>
Date:   2018-05-08T10:06:47Z

    [CARBONDATA-2455]To support TEXT_MATCH to search empty string with Wildcard(*) and AND,OR,NOT Filter fix

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2281
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5731/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2281
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4570/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2281
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5740/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2281
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4801/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2281
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4803/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2281
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4584/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2281
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5754/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2281
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4595/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2281
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4815/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2281
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4841/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2281#discussion_r187257103
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMapSuite.scala ---
    @@ -733,6 +732,26 @@ class LuceneFineGrainDataMapSuite extends QueryTest with BeforeAndAfterAll {
         sql("DROP TABLE table1")
       }
     
    +  test("test lucene fine grain empty string") {
    +    sql("DROP TABLE IF EXISTS datamap_test_01")
    +    CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_LUCENE_EMPTY_STRING_ENABLE,"null")
    +    sql(
    +      "create table datamap_test_01(id int,fn string,ln string)stored by 'carbondata' " +
    --- End diff --
   
    use `stored as carbondata` and move to next line. please make it nice formated and captital


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2281#discussion_r187257562
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMapSuite.scala ---
    @@ -733,6 +732,26 @@ class LuceneFineGrainDataMapSuite extends QueryTest with BeforeAndAfterAll {
         sql("DROP TABLE table1")
       }
     
    +  test("test lucene fine grain empty string") {
    +    sql("DROP TABLE IF EXISTS datamap_test_01")
    +    CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_LUCENE_EMPTY_STRING_ENABLE,"null")
    +    sql(
    +      "create table datamap_test_01(id int,fn string,ln string)stored by 'carbondata' " +
    +      "tblproperties('sort_columns'='fn','sort_scope'='local_sort')")
    +    sql(
    +      s"""
    +         | CREATE DATAMAP dm_datamap ON TABLE datamap_test_01
    +         | USING 'lucene'
    +         | DMProperties('INDEX_COLUMNS'='fn,ln')
    +      """.stripMargin)
    +    sql("insert into table datamap_test_01 select 1,'abc',''")
    +    sql("insert into table datamap_test_01 select 2,'abc def','ghi'")
    +    checkAnswer(sql("select *from datamap_test_01 where ln like '%'"),
    +      sql("select *from datamap_test_01 where text_match('ln:*')"))
    --- End diff --
   
    In current master, what is the result set for this query? I think it is acceptable that result is different from non-lucene table query, since lucene has its own behavior. Our goal is to comply to lucene behavior but not make everything the same as like query


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2281#discussion_r187257831
 
    --- Diff: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/LuceneTestCase.scala ---
    @@ -110,14 +110,13 @@ class LuceneTestCase extends QueryTest with BeforeAndAfterAll {
         checkAnswer(sql("SELECT * FROM datamap_main WHERE TEXT_MATCH('country:ch*')"),
           sql("select * from datamap_main where country like 'ch%'"))
         checkAnswer(sql(
    -      "SELECT * FROM datamap_main WHERE TEXT_MATCH('country:ch*') AND TEXT_MATCH('name:aa*')"),
    +      "SELECT * FROM datamap_main WHERE TEXT_MATCH('country:ch* AND name:aa*')"),
           sql("select * from datamap_main where country like 'ch%' and name like 'aa%'"))
         checkAnswer(sql(
    -      "SELECT * FROM datamap_main WHERE TEXT_MATCH('country:u* or name:aa*')"),
    +      "SELECT * FROM datamap_main WHERE TEXT_MATCH('country:u* OR name:aa*')"),
    --- End diff --
   
    you mean to say `country:u* or name:aa*` is different from `country:u* OR name:aa*`?


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2281#discussion_r187258220
 
    --- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneDataMapWriter.java ---
    @@ -278,6 +278,16 @@ private boolean addField(Document doc, Object data, CarbonColumn column, Field.S
           String strValue = null;
    --- End diff --
   
    move previous TODO


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2281
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5788/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2281
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4633/



---

[qiuchenjian-2]

Github user Indhumathi27 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2281#discussion_r187269147
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMapSuite.scala ---
    @@ -733,6 +732,26 @@ class LuceneFineGrainDataMapSuite extends QueryTest with BeforeAndAfterAll {
         sql("DROP TABLE table1")
       }
     
    +  test("test lucene fine grain empty string") {
    +    sql("DROP TABLE IF EXISTS datamap_test_01")
    +    CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_LUCENE_EMPTY_STRING_ENABLE,"null")
    +    sql(
    +      "create table datamap_test_01(id int,fn string,ln string)stored by 'carbondata' " +
    --- End diff --
   
    okay


---

[qiuchenjian-2]

Github user Indhumathi27 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2281#discussion_r187269681
 
    --- Diff: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/LuceneTestCase.scala ---
    @@ -110,14 +110,13 @@ class LuceneTestCase extends QueryTest with BeforeAndAfterAll {
         checkAnswer(sql("SELECT * FROM datamap_main WHERE TEXT_MATCH('country:ch*')"),
           sql("select * from datamap_main where country like 'ch%'"))
         checkAnswer(sql(
    -      "SELECT * FROM datamap_main WHERE TEXT_MATCH('country:ch*') AND TEXT_MATCH('name:aa*')"),
    +      "SELECT * FROM datamap_main WHERE TEXT_MATCH('country:ch* AND name:aa*')"),
           sql("select * from datamap_main where country like 'ch%' and name like 'aa%'"))
         checkAnswer(sql(
    -      "SELECT * FROM datamap_main WHERE TEXT_MATCH('country:u* or name:aa*')"),
    +      "SELECT * FROM datamap_main WHERE TEXT_MATCH('country:u* OR name:aa*')"),
    --- End diff --
   
    yes..For lucene -standard analyzer ,filters should be given in UPPER case


---

[qiuchenjian-2]

Github user Indhumathi27 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2281#discussion_r187273835
 
    --- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneDataMapWriter.java ---
    @@ -278,6 +278,16 @@ private boolean addField(Document doc, Object data, CarbonColumn column, Field.S
           String strValue = null;
    --- End diff --
   
    okay


---