[GitHub] carbondata pull request #2491: [CARBONDATA-2700][BloomDataMap] Block droppin...

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r202000213
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonCreateDataMapCommand.scala ---
    @@ -109,6 +111,19 @@ case class CarbonCreateDataMapCommand(
                 throw new MalformedDataMapCommandException(String.format(
                   "column '%s' already has %s index datamap created",
                   column.getColName, thisDmProviderName))
    +          } else if (isBloomFilter) {
    +            // if datamap provider is bloomfilter,the index column datatype cannot be complex type
    +            if (column.isComplex) {
    +              throw new MalformedDataMapCommandException(String.format(
    +                "since datamap provider is bloomfilter , the index column '%s' datatype cannot be" +
    --- End diff --
   
    optimize the message format like "BloomFilter datamap does not support complex datatype column: ${column.getColName}"
   
    The same for the below if branch


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r202000492
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableDropColumnCommand.scala ---
    @@ -58,7 +58,10 @@ private[sql] case class CarbonAlterTableDropColumnCommand(
             .validateTableAndAcquireLock(dbName, tableName, locksToBeAcquired)(sparkSession)
           val metastore = CarbonEnv.getInstance(sparkSession).carbonMetastore
           carbonTable = CarbonEnv.getCarbonTable(Some(dbName), tableName)(sparkSession)
    -      if (!carbonTable.canAllow(carbonTable, TableOperation.ALTER_DROP)) {
    +      if (!carbonTable
    --- End diff --
   
    please optimize the indent


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r202000664
 
    --- Diff: integration/spark2/src/test/scala/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMapSuite.scala ---
    @@ -377,7 +382,91 @@ class BloomCoarseGrainDataMapSuite extends QueryTest with BeforeAndAfterAll with
         checkQuery("fakeDm", shouldHit = false)
       }
     
    +  test("test blocl change datatype for bloomfilter index datamap") {
    --- End diff --
   
    block?


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r201999690
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala ---
    @@ -302,13 +302,20 @@ abstract class CarbonDDLSqlParser extends AbstractCarbonSparkSQLParser {
           if (!CarbonScalaUtil
             .validateLocalDictionaryEnable(tableProperties(CarbonCommonConstants
    --- End diff --
   
    optimize the indent, move 'validateLocalDictionaryEnable' to previous line


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r202000836
 
    --- Diff: integration/spark2/src/test/scala/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMapSuite.scala ---
    @@ -377,7 +382,91 @@ class BloomCoarseGrainDataMapSuite extends QueryTest with BeforeAndAfterAll with
         checkQuery("fakeDm", shouldHit = false)
       }
     
    +  test("test blocl change datatype for bloomfilter index datamap") {
    +    sql(
    +      s"""
    +         | CREATE TABLE $normalTable(id INT, name STRING, city STRING, age INT,
    +         | s1 STRING, s2 STRING, s3 STRING, s4 STRING, s5 STRING, s6 STRING, s7 STRING, s8 STRING)
    +         | STORED BY 'carbondata' TBLPROPERTIES('table_blocksize'='128')
    +         | """.stripMargin)
    +
    +    sql(
    +      s"""
    +         | CREATE DATAMAP $dataMapName ON TABLE $normalTable
    +         | USING 'bloomfilter' WITH DEFERRED REBUILD
    +         | DMProperties( 'INDEX_COLUMNS'='city,id', 'BLOOM_SIZE'='640000')
    +      """.stripMargin)
    +    val exception: MalformedCarbonCommandException = intercept[MalformedCarbonCommandException] {
    +      sql(s"ALTER TABLE $normalTable CHANGE id id bigint")
    +    }
    +    assert(exception.getMessage
    +      .contains("alter table change datatype is not supported for index datamap"))
    +  }
    +
    +  test("test drop index columns for bloomfilter datamap") {
    +    sql(
    +      s"""
    +         | CREATE TABLE $normalTable(id INT, name STRING, city STRING, age INT,
    +         | s1 STRING, s2 STRING, s3 STRING, s4 STRING, s5 STRING, s6 STRING, s7 STRING, s8 STRING)
    +         | STORED BY 'carbondata' TBLPROPERTIES('table_blocksize'='128')
    +         |  """.stripMargin)
    +    sql(
    +      s"""
    +         | CREATE DATAMAP $dataMapName ON TABLE $normalTable
    +         | USING 'bloomfilter'
    +         | WITH DEFERRED REBUILD
    +         | DMProperties('INDEX_COLUMNS'='city,id', 'BLOOM_SIZE'='640000')
    +      """.stripMargin)
    +    val exception: MalformedCarbonCommandException = intercept[MalformedCarbonCommandException] {
    +      sql(s"alter table $normalTable drop columns(name, id)")
    +    }
    +    assert(exception.getMessage
    +      .contains("alter table drop column is not supported for index datamap"))
    +  }
    +
    +  test("test create bloomfilter datamap which index column is localdictcolumn ") {
    +    sql(
    +      s"""
    +         | CREATE TABLE $normalTable(id INT, name STRING, city STRING, age INT,
    +         | s1 STRING, s2 STRING, s3 STRING, s4 STRING, s5 STRING, s6 STRING, s7 STRING, s8 STRING)
    +         | STORED BY 'carbondata' TBLPROPERTIES('local_dictionary_enable'='true',
    +         | 'local_dictionary_include'='city')
    +         |  """.stripMargin)
    +    val exception: MalformedDataMapCommandException = intercept[MalformedDataMapCommandException] {
    +      sql(
    +        s"""
    +           | CREATE DATAMAP $dataMapName ON TABLE $normalTable
    +           | USING 'bloomfilter'
    +           | WITH DEFERRED REBUILD
    +           | DMProperties('INDEX_COLUMNS'='city,id', 'BLOOM_SIZE'='640000')
    +      """.stripMargin)
    +    }
    +    assert(exception.getMessage
    +      .contains("cannot be localDictColumn"))
    +  }
    +
    +  test("test create bloomfilter datamap which index column datatype is complex ") {
    +    sql(
    +      s"""
    +         | CREATE TABLE $normalTable(id INT, name STRING, city Array<INT>, age INT,
    +         | s1 STRING, s2 STRING, s3 STRING, s4 STRING, s5 STRING, s6 STRING, s7 STRING, s8 STRING)
    +         | STORED BY 'carbondata'
    +         |  """.stripMargin)
    +    val exception: MalformedDataMapCommandException = intercept[MalformedDataMapCommandException] {
    +      sql(
    +        s"""
    +           | CREATE DATAMAP $dataMapName ON TABLE $normalTable
    +           | USING 'bloomfilter'
    +           | WITH DEFERRED REBUILD
    +           | DMProperties('INDEX_COLUMNS'='city,id', 'BLOOM_SIZE'='640000')
    +      """.stripMargin)
    +    }
    +    assert(exception.getMessage
    --- End diff --
   
    please optimize the indent


---

[qiuchenjian-2]

Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/2491
 
    please check and fix the comments


---

[qiuchenjian-2]

Github user Sssan520 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r202221482
 
    --- Diff: datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMapFactory.java ---
    @@ -380,6 +380,49 @@ public void deleteDatamapData() {
         return false;
       }
     
    +  @Override public boolean isOperationBlock(TableOperation operation, Object... targets) {
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user Sssan520 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r202221610
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/DataMapFactory.java ---
    @@ -144,4 +144,17 @@ public void validate() throws MalformedDataMapCommandException {
         }
       }
     
    +  /**
    +   * to check something like columns by the operation
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user Sssan520 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r202221623
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java ---
    @@ -1058,9 +1058,10 @@ public void setTransactionalTable(boolean transactionalTable) {
        * if this operation makes datamap stale it is not allowed
        * @param carbonTable
        * @param operation
    +   * @param targets
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user Sssan520 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r202221696
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java ---
    @@ -1071,6 +1072,10 @@ public boolean canAllow(CarbonTable carbonTable, TableOperation operation) {
               if (factoryClass.willBecomeStale(operation)) {
                 return false;
               }
    +          //to check something like columns by the operation
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user Sssan520 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r202221707
 
    --- Diff: datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMapFactory.java ---
    @@ -380,6 +380,49 @@ public void deleteDatamapData() {
         return false;
       }
     
    +  @Override public boolean isOperationBlock(TableOperation operation, Object... targets) {
    +    switch (operation) {
    +      case ALTER_DROP: {
    +        try {
    +          //alter table drop columns
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user Sssan520 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r202221774
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonCreateDataMapCommand.scala ---
    @@ -109,6 +111,19 @@ case class CarbonCreateDataMapCommand(
                 throw new MalformedDataMapCommandException(String.format(
                   "column '%s' already has %s index datamap created",
                   column.getColName, thisDmProviderName))
    +          } else if (isBloomFilter) {
    +            // if datamap provider is bloomfilter,the index column datatype cannot be complex type
    +            if (column.isComplex) {
    +              throw new MalformedDataMapCommandException(String.format(
    +                "since datamap provider is bloomfilter , the index column '%s' datatype cannot be" +
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user Sssan520 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r202221798
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableDataTypeChangeCommand.scala ---
    @@ -55,7 +55,10 @@ private[sql] case class CarbonAlterTableDataTypeChangeCommand(
             .validateTableAndAcquireLock(dbName, tableName, locksToBeAcquired)(sparkSession)
           val metastore = CarbonEnv.getInstance(sparkSession).carbonMetastore
           carbonTable = CarbonEnv.getCarbonTable(Some(dbName), tableName)(sparkSession)
    -      if (!carbonTable.canAllow(carbonTable, TableOperation.ALTER_CHANGE_DATATYPE)) {
    +      if (!carbonTable
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user Sssan520 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r202226296
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableDropColumnCommand.scala ---
    @@ -58,7 +58,10 @@ private[sql] case class CarbonAlterTableDropColumnCommand(
             .validateTableAndAcquireLock(dbName, tableName, locksToBeAcquired)(sparkSession)
           val metastore = CarbonEnv.getInstance(sparkSession).carbonMetastore
           carbonTable = CarbonEnv.getCarbonTable(Some(dbName), tableName)(sparkSession)
    -      if (!carbonTable.canAllow(carbonTable, TableOperation.ALTER_DROP)) {
    +      if (!carbonTable
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user Sssan520 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2491#discussion_r202228474
 
    --- Diff: integration/spark2/src/test/scala/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMapSuite.scala ---
    @@ -377,7 +382,91 @@ class BloomCoarseGrainDataMapSuite extends QueryTest with BeforeAndAfterAll with
         checkQuery("fakeDm", shouldHit = false)
       }
     
    +  test("test blocl change datatype for bloomfilter index datamap") {
    --- End diff --
   
    yes


---

[qiuchenjian-2]

Github user Sssan520 commented on the issue:

    https://github.com/apache/carbondata/pull/2491
 
    all code review comments has been handled.


---

[qiuchenjian-2]

Github user Sssan520 commented on the issue:

    https://github.com/apache/carbondata/pull/2491
 
    retest this please


---

[qiuchenjian-2]

Github user Sssan520 closed the pull request at:

    https://github.com/apache/carbondata/pull/2491


---

[qiuchenjian-2]

GitHub user Sssan520 reopened a pull request:

    https://github.com/apache/carbondata/pull/2491

    [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2730][CARBONDATA-2732][BloomDataMap] block some oprerations of bloomfilter datamap

    1.Block create bloomfilter datamap index on column which its datatype is complex type;
    2.Block change datatype for bloomfilter index datamap;
    3.Block dropping index columns for bloomfilter index datamap
   
    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed?
     add a parameter "targets" for method "canAllow" of "CarbonTable" class
   
     - [ ] Any backward compatibility impacted?  No
     
     - [ ] Document update required? No
   
     - [ ] Testing done NA
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Sssan520/carbondata bloomfilter

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2491.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2491
   
----
commit 815c497380fa52b98aa270d35ce67f2cb9c67191
Author: lianganping 00251374 <lianganping@...>
Date:   2018-07-11T12:17:09Z

    1.Block create bloomfilter datamap index on column which its datatype is complex;2.Block create bloomfilter datamap index on local_dictionary column;3.Block change datatype for bloomfilter index datamap;4.Block dropping index columns for bloomfilter index datamap

----


---

[qiuchenjian-2]

Github user Sssan520 closed the pull request at:

    https://github.com/apache/carbondata/pull/2491


---