[GitHub] carbondata pull request #2422: [CARBONDATA-2585][CARBONDATA-2586][Local Dict...

[qiuchenjian-2]

GitHub user brijoobopanna opened a pull request:

    https://github.com/apache/carbondata/pull/2422

    [CARBONDATA-2585][CARBONDATA-2586][Local Dictionary]Added test cases for local dictionary support for alter table, set, unset and preaggregate

    Dependent on #2401
   
    **What changes were proposed in this pull request?**
   
    In this PR,
    Added test cases for local dictionary support for alter table, set, unset and pre-aggregate
    All the validations related to above features are taken care in this PR
   
    **How was this patch tested?**
    Test case were executed on a 3 node cluster and the same were automated and added as SDV test cases

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/brijoobopanna/carbondata FT_alter_set_unset

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2422.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2422
   
----
commit 730540ded585fbac997d2f18538efc54fc82c374
Author: akashrn5 <akashnilugal@...>
Date:   2018-06-22T15:40:13Z

    Local dictionary support for alter table, preaggregate,varchar datatype, set and unset command

commit e6ec12084b1af49f615f80eb6cd68eba99d2bfe5
Author: brijoobopanna <brijoobopanna@...>
Date:   2018-06-27T09:22:46Z

    Added test cases for local dictionary support for alter table,set, unset and preaggregate

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2422
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6588/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2422
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5412/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2422
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5481/



---

[qiuchenjian-2]

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2422
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2422
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6606/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2422
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5434/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2422
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6649/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2422
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5476/



---

[qiuchenjian-2]

Github user kunal642 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2422#discussion_r199126014
 
    --- Diff: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/CreateTableWithLocalDictionaryTestCase.scala ---
    @@ -2096,6 +2096,264 @@ class CreateTableWithLocalDictionaryTestCase extends QueryTest with BeforeAndAft
         }
       }
     
    +  test("test alter table add column") {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'org.apache.carbondata.format' tblproperties('local_dictionary_enable'='true',
    +        | 'local_dictionary_threshold'='20000','local_dictionary_include'='city','no_inverted_index'='name')
    +      """.stripMargin)
    +    sql("alter table local1 add columns (alt string) tblproperties('local_dictionary_include'='alt')")
    +    val descLoc = sql("describe formatted local1").collect
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("20000"))
    +    }
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("true"))
    +    }
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city,alt"))
    +    }
    +  }
    +
    +  test("test alter table add column default configs for local dictionary") {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'org.apache.carbondata.format' tblproperties('local_dictionary_enable'='true',
    +        | 'local_dictionary_threshold'='20000','no_inverted_index'='name')
    +      """.stripMargin)
    +    sql("alter table local1 add columns (alt string)")
    +    val descLoc = sql("describe formatted local1").collect
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("20000"))
    +    }
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("true"))
    +    }
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("name,city,alt"))
    +    }
    +  }
    +
    +  test("test alter table add column where same column is in dictionary include and local dictionary include") {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'org.apache.carbondata.format' tblproperties('local_dictionary_enable'='true',
    +        | 'local_dictionary_threshold'='20000','local_dictionary_include'='city','no_inverted_index'='name')
    +      """.stripMargin)
    +    val exception = intercept[MalformedCarbonCommandException] {
    +      sql(
    +        "alter table local1 add columns (alt string) tblproperties('local_dictionary_include'='alt','dictionary_include'='alt')")
    +    }
    +    assert(exception.getMessage
    +      .contains(
    +        "LOCAL_DICTIONARY_INCLUDE/LOCAL_DICTIONARY_EXCLUDE column: alt specified in Dictionary " +
    +        "include. Local Dictionary will not be generated for Dictionary include columns. Please " +
    +        "check create table statement."))
    +  }
    +
    +  test("test alter table add column where duplicate columns present in local dictionary include") {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'org.apache.carbondata.format' tblproperties('local_dictionary_enable'='true',
    +        | 'local_dictionary_threshold'='20000','local_dictionary_include'='city','no_inverted_index'='name')
    +      """.stripMargin)
    +    val exception = intercept[MalformedCarbonCommandException] {
    +      sql(
    +        "alter table local1 add columns (alt string) tblproperties('local_dictionary_include'='alt,alt')")
    +    }
    +    assert(exception.getMessage
    +      .contains(
    +        "LOCAL_DICTIONARY_INCLUDE/LOCAL_DICTIONARY_EXCLUDE contains Duplicate Columns: alt. " +
    +        "Please check create table statement."))
    +  }
    +
    +  test("test alter table add column where duplicate columns present in local dictionary include/exclude")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'org.apache.carbondata.format' tblproperties('local_dictionary_enable'='true',
    +        | 'local_dictionary_threshold'='20000','local_dictionary_include'='city',
    +        | 'no_inverted_index'='name')
    +      """.stripMargin)
    +    val exception1 = intercept[MalformedCarbonCommandException] {
    +      sql(
    +        "alter table local1 add columns (alt string) tblproperties" +
    +        "('local_dictionary_include'='abc')")
    +    }
    +    assert(exception1.getMessage
    +      .contains(
    +        "LOCAL_DICTIONARY_INCLUDE/LOCAL_DICTIONARY_EXCLUDE column: abc does not exist in table. " +
    +        "Please check create table statement."))
    +    val exception2 = intercept[MalformedCarbonCommandException] {
    +      sql(
    +        "alter table local1 add columns (alt string) tblproperties" +
    +        "('local_dictionary_exclude'='abc')")
    +    }
    +    assert(exception2.getMessage
    +      .contains(
    +        "LOCAL_DICTIONARY_INCLUDE/LOCAL_DICTIONARY_EXCLUDE column: abc does not exist in table. " +
    +        "Please check create table statement."))
    +  }
    +
    +  test("test alter table add column for datatype validation")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """ | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'org.apache.carbondata.format' tblproperties('local_dictionary_enable'='true',
    +        | 'local_dictionary_include'='city', 'no_inverted_index'='name')
    +      """.stripMargin)
    +    val exception = intercept[MalformedCarbonCommandException] {
    +      sql(
    +        "alter table local1 add columns (alt string,abc int) tblproperties" +
    +        "('local_dictionary_include'='abc')")
    +    }
    +    assert(exception.getMessage
    +      .contains(
    +        "LOCAL_DICTIONARY_INCLUDE/LOCAL_DICTIONARY_EXCLUDE column: abc is not a String/complex " +
    +        "datatype column. LOCAL_DICTIONARY_COLUMN should be no dictionary string/complex datatype" +
    +        " column.Please check create table statement."))
    +  }
    +
    +  test("test alter table add column where duplicate columns are present in local dictionary include and exclude")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'org.apache.carbondata.format' tblproperties('local_dictionary_enable'='true',
    +        | 'local_dictionary_include'='city', 'no_inverted_index'='name')
    +      """.stripMargin)
    +    val exception = intercept[MalformedCarbonCommandException] {
    +      sql(
    +        "alter table local1 add columns (alt string,abc string) tblproperties" +
    +        "('local_dictionary_include'='abc','local_dictionary_exclude'='alt,abc')")
    +    }
    +    assert(exception.getMessage
    +      .contains(
    +        "Column ambiguity as duplicate column(s):abc is present in LOCAL_DICTIONARY_INCLUDE " +
    +        "and LOCAL_DICTIONARY_EXCLUDE. Duplicate columns are not allowed."))
    +  }
    +
    +  test("test alter table add column unsupported table property")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'org.apache.carbondata.format' tblproperties('local_dictionary_enable'='true',
    +        | 'local_dictionary_include'='city', 'no_inverted_index'='name')
    +      """.stripMargin)
    +    val exception = intercept[MalformedCarbonCommandException] {
    +      sql(
    +        "alter table local1 add columns (alt string,abc string) tblproperties" +
    +        "('local_dictionary_enable'='abc')")
    +    }
    +    assert(exception.getMessage
    +      .contains(
    +        "Unsupported Table property in add column: local_dictionary_enable"))
    +    val exception1 = intercept[MalformedCarbonCommandException] {
    +      sql(
    +        "alter table local1 add columns (alt string,abc string) tblproperties" +
    +        "('local_dictionary_threshold'='10000')")
    +    }
    +    assert(exception1.getMessage
    +      .contains(
    +        "Unsupported Table property in add column: local_dictionary_threshold"))
    +  }
    +
    +  test("test alter table add column when main table is disabled for local dictionary")
    --- End diff --
   
    Add one test case in which 2 string columns are added using alter but only one column is specified in local_dic_include. Both should be considered for local dict generation.


---

[qiuchenjian-2]

Github user kunal642 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2422#discussion_r199126397
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/localdictionary/LocalDictionarySupportCreateTableTest.scala ---
    @@ -2585,67 +2585,445 @@ class LocalDictionarySupportCreateTableTest extends QueryTest with BeforeAndAfte
         }
       }
     
    -  test("test preaggregate table local dictionary enabled table")
    +  test("test local dictionary foer varchar datatype columns") {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'org.apache.carbondata.format' tblproperties('local_dictionary_include'='city',
    +        | 'LONG_STRING_COLUMNS'='city')
    +      """.stripMargin)
    +    val descLoc = sql("describe formatted local1").collect
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city"))
    +    }
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("10000"))
    +    }
    +  }
    +
    +  test("test local dictionary describe formatted only with default configs")
       {
         sql("drop table if exists local1")
    -    sql("CREATE TABLE local1 (id Int, date date, country string, phonetype string, " +
    -        "serialname String,salary int ) STORED BY 'org.apache.carbondata.format' " +
    -        "tblproperties('dictionary_include'='country','local_dictionary_enable'='true','local_dictionary_include' = 'phonetype','local_dictionary_exclude' ='serialname')")
    -    sql("create datamap PreAggCount on table local1 using 'preaggregate' as " +
    -        "select country,count(salary) as count from local1 group by country")
    -    val descLoc = sql("describe formatted local1_PreAggCount").collect
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'carbondata'
    +      """.stripMargin)
    +
    +    val descLoc = sql("describe formatted local1").collect
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("true"))
    +    }
         descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
           case Some(row) => assert(row.get(1).toString.contains("10000"))
         }
         descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    -      case Some(row) => assert(row.get(1).toString.contains("phonetype"))
    +      case Some(row) => assert(row.get(1).toString.contains("name,city"))
         }
    -    descLoc.find(_.get(0).toString.contains("Local Dictionary Exclude")) match {
    -      case Some(row) => assert(row.get(1).toString.contains("serialname"))
    +  }
    +
    +  test("test local dictionary for invalid threshold")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_threshold'='300000')
    +      """.stripMargin)
    +
    +    val descLoc = sql("describe formatted local1").collect
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("10000"))
         }
    -    descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +  }
    +
    +  test("test alter set for local dictionary enable to disable")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_threshold'='300000')
    +      """.stripMargin)
    +
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
           case Some(row) => assert(row.get(1).toString.contains("true"))
         }
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("10000"))
    +    }
    +    sql("alter table local1 set tblproperties('local_dictionary_enable'='false')")
    +    val descLoc2 = sql("describe formatted local1").collect
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("false"))
    +    }
    +    checkExistence(sql("DESC FORMATTED local1"), false,
    +      "Local Dictionary Threshold")
       }
     
    -  test("test local dictionary foer varchar datatype columns") {
    +  test("test alter set for local dictionary disable to enable")
    +  {
         sql("drop table if exists local1")
         sql(
           """
             | CREATE TABLE local1(id int, name string, city string, age int)
    -        | STORED BY 'org.apache.carbondata.format' tblproperties('local_dictionary_include'='city',
    -        | 'LONG_STRING_COLUMNS'='city')
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_enable'='false','local_dictionary_threshold'='300000')
           """.stripMargin)
    -    val descLoc = sql("describe formatted local1").collect
    -    descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    -      case Some(row) => assert(row.get(1).toString.contains("city"))
    +
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("false"))
         }
    -    descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +    checkExistence(sql("DESC FORMATTED local1"), false,
    +      "Local Dictionary Threshold")
    +    sql("alter table local1 set tblproperties('local_dictionary_enable'='true','local_dictionary_threshold'='30000')")
    +    val descLoc2 = sql("describe formatted local1").collect
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("true"))
    +    }
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("30000"))
    +    }
    +  }
    +
    +  test("test alter set for local dictionary threshold valid and invalid")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_threshold'='20000')
    +      """.stripMargin)
    +
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("20000"))
    +    }
    +    sql("alter table local1 set tblproperties('local_dictionary_threshold'='30000')")
    +    val descLoc2 = sql("describe formatted local1").collect
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("30000"))
    +    }
    +    sql("alter table local1 set tblproperties('local_dictionary_threshold'='500000')")
    +    val descLoc3 = sql("describe formatted local1").collect
    +    descLoc3.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
           case Some(row) => assert(row.get(1).toString.contains("10000"))
         }
       }
     
    -  test("test local dictionary describe formatted only with default configs")
    +  test("test alter set for local dictionary include valid and invalid")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int,add string)
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_include'='city','dictionary_include'='add')
    +      """.stripMargin)
    +
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city"))
    +    }
    +    sql("alter table local1 set tblproperties('local_dictionary_include'='name')")
    +    val descLoc2 = sql("describe formatted local1").collect
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("name"))
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_include'='id')")
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_include'='city,city')")
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_include'='')")
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_include'='add')")
    +    }
    +  }
    +
    +  test("test alter set for local dictionary exclude valid and invalid")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int,add string)
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_exclude'='city','dictionary_include'='add')
    +      """.stripMargin)
    +
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Exclude")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city"))
    +    }
    +    sql("alter table local1 set tblproperties('local_dictionary_exclude'='name')")
    +    val descLoc2 = sql("describe formatted local1").collect
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Exclude")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("name"))
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_exclude'='id')")
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_exclude'='city,city')")
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_exclude'='')")
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_exclude'='add')")
    +    }
    +  }
    +
    +  test("test alter set same column for local dictionary exclude and include")
       {
         sql("drop table if exists local1")
         sql(
           """
             | CREATE TABLE local1(id int, name string, city string, age int)
    -        | STORED BY 'carbondata'
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_exclude'='city')
           """.stripMargin)
    +    intercept[Exception] {
    +      sql(
    +        "alter table local1 set tblproperties('local_dictionary_include'='name'," +
    +        "'local_dictionary_exclude'='name')")
    +    }
    +  }
     
    +  test("test alter set for valid and invalid complex type as include/exclude") {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int,st struct<s_id:int,
    +        | s_name:string,s_city:array<string>>, dcity array<string>)
    +        | STORED BY 'org.apache.carbondata.format'
    +        | tblproperties('local_dictionary_exclude'='name','local_dictionary_include'='city,dcity',
    +        | 'local_dictionary_enable'='true')
    +      """.
    +        stripMargin)
    +    val descFormatted1 = sql("describe formatted local1").collect
    +    descFormatted1.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("true"))
    +    }
    +    descFormatted1.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city,dcity"))
    --- End diff --
   
    add check to verify that name is not included in local dictionary include property.


---

[qiuchenjian-2]

Github user kunal642 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2422#discussion_r199126782
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/localdictionary/LocalDictionarySupportCreateTableTest.scala ---
    @@ -2585,67 +2585,445 @@ class LocalDictionarySupportCreateTableTest extends QueryTest with BeforeAndAfte
         }
       }
     
    -  test("test preaggregate table local dictionary enabled table")
    +  test("test local dictionary foer varchar datatype columns") {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'org.apache.carbondata.format' tblproperties('local_dictionary_include'='city',
    +        | 'LONG_STRING_COLUMNS'='city')
    +      """.stripMargin)
    +    val descLoc = sql("describe formatted local1").collect
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city"))
    +    }
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("10000"))
    +    }
    +  }
    +
    +  test("test local dictionary describe formatted only with default configs")
       {
         sql("drop table if exists local1")
    -    sql("CREATE TABLE local1 (id Int, date date, country string, phonetype string, " +
    -        "serialname String,salary int ) STORED BY 'org.apache.carbondata.format' " +
    -        "tblproperties('dictionary_include'='country','local_dictionary_enable'='true','local_dictionary_include' = 'phonetype','local_dictionary_exclude' ='serialname')")
    -    sql("create datamap PreAggCount on table local1 using 'preaggregate' as " +
    -        "select country,count(salary) as count from local1 group by country")
    -    val descLoc = sql("describe formatted local1_PreAggCount").collect
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'carbondata'
    +      """.stripMargin)
    +
    +    val descLoc = sql("describe formatted local1").collect
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("true"))
    +    }
         descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
           case Some(row) => assert(row.get(1).toString.contains("10000"))
         }
         descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    -      case Some(row) => assert(row.get(1).toString.contains("phonetype"))
    +      case Some(row) => assert(row.get(1).toString.contains("name,city"))
         }
    -    descLoc.find(_.get(0).toString.contains("Local Dictionary Exclude")) match {
    -      case Some(row) => assert(row.get(1).toString.contains("serialname"))
    +  }
    +
    +  test("test local dictionary for invalid threshold")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_threshold'='300000')
    +      """.stripMargin)
    +
    +    val descLoc = sql("describe formatted local1").collect
    +    descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("10000"))
         }
    -    descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +  }
    +
    +  test("test alter set for local dictionary enable to disable")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_threshold'='300000')
    +      """.stripMargin)
    +
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
           case Some(row) => assert(row.get(1).toString.contains("true"))
         }
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("10000"))
    +    }
    +    sql("alter table local1 set tblproperties('local_dictionary_enable'='false')")
    +    val descLoc2 = sql("describe formatted local1").collect
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("false"))
    +    }
    +    checkExistence(sql("DESC FORMATTED local1"), false,
    +      "Local Dictionary Threshold")
       }
     
    -  test("test local dictionary foer varchar datatype columns") {
    +  test("test alter set for local dictionary disable to enable")
    +  {
         sql("drop table if exists local1")
         sql(
           """
             | CREATE TABLE local1(id int, name string, city string, age int)
    -        | STORED BY 'org.apache.carbondata.format' tblproperties('local_dictionary_include'='city',
    -        | 'LONG_STRING_COLUMNS'='city')
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_enable'='false','local_dictionary_threshold'='300000')
           """.stripMargin)
    -    val descLoc = sql("describe formatted local1").collect
    -    descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    -      case Some(row) => assert(row.get(1).toString.contains("city"))
    +
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("false"))
         }
    -    descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +    checkExistence(sql("DESC FORMATTED local1"), false,
    +      "Local Dictionary Threshold")
    +    sql("alter table local1 set tblproperties('local_dictionary_enable'='true','local_dictionary_threshold'='30000')")
    +    val descLoc2 = sql("describe formatted local1").collect
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("true"))
    +    }
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("30000"))
    +    }
    +  }
    +
    +  test("test alter set for local dictionary threshold valid and invalid")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int)
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_threshold'='20000')
    +      """.stripMargin)
    +
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("20000"))
    +    }
    +    sql("alter table local1 set tblproperties('local_dictionary_threshold'='30000')")
    +    val descLoc2 = sql("describe formatted local1").collect
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("30000"))
    +    }
    +    sql("alter table local1 set tblproperties('local_dictionary_threshold'='500000')")
    +    val descLoc3 = sql("describe formatted local1").collect
    +    descLoc3.find(_.get(0).toString.contains("Local Dictionary Threshold")) match {
           case Some(row) => assert(row.get(1).toString.contains("10000"))
         }
       }
     
    -  test("test local dictionary describe formatted only with default configs")
    +  test("test alter set for local dictionary include valid and invalid")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int,add string)
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_include'='city','dictionary_include'='add')
    +      """.stripMargin)
    +
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city"))
    +    }
    +    sql("alter table local1 set tblproperties('local_dictionary_include'='name')")
    +    val descLoc2 = sql("describe formatted local1").collect
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("name"))
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_include'='id')")
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_include'='city,city')")
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_include'='')")
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_include'='add')")
    +    }
    +  }
    +
    +  test("test alter set for local dictionary exclude valid and invalid")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int,add string)
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_exclude'='city','dictionary_include'='add')
    +      """.stripMargin)
    +
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Exclude")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city"))
    +    }
    +    sql("alter table local1 set tblproperties('local_dictionary_exclude'='name')")
    +    val descLoc2 = sql("describe formatted local1").collect
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Exclude")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("name"))
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_exclude'='id')")
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_exclude'='city,city')")
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_exclude'='')")
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_exclude'='add')")
    +    }
    +  }
    +
    +  test("test alter set same column for local dictionary exclude and include")
       {
         sql("drop table if exists local1")
         sql(
           """
             | CREATE TABLE local1(id int, name string, city string, age int)
    -        | STORED BY 'carbondata'
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_exclude'='city')
           """.stripMargin)
    +    intercept[Exception] {
    +      sql(
    +        "alter table local1 set tblproperties('local_dictionary_include'='name'," +
    +        "'local_dictionary_exclude'='name')")
    +    }
    +  }
     
    +  test("test alter set for valid and invalid complex type as include/exclude") {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int,st struct<s_id:int,
    +        | s_name:string,s_city:array<string>>, dcity array<string>)
    +        | STORED BY 'org.apache.carbondata.format'
    +        | tblproperties('local_dictionary_exclude'='name','local_dictionary_include'='city,dcity',
    +        | 'local_dictionary_enable'='true')
    +      """.
    +        stripMargin)
    +    val descFormatted1 = sql("describe formatted local1").collect
    +    descFormatted1.find(_.get(0).toString.contains("Local Dictionary Enabled")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("true"))
    +    }
    +    descFormatted1.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city,dcity"))
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_exclude'='dcity')")
    +    }
    +    sql("alter table local1 set tblproperties('local_dictionary_exclude'='st')")
    +    val descFormatted2 = sql("describe formatted local1").collect
    +    descFormatted2.find(_.get(0).toString.contains("Local Dictionary Exclude")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("st"))
    +    }
    +    descFormatted2.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city,dcity"))
    +    }
    +    sql("alter table local1 set tblproperties('local_dictionary_exclude'='st'," +
    +        "'local_dictionary_include'='dcity')")
    +    val descFormatted3 = sql("describe formatted local1").collect
    +    descFormatted3.find(_.get(0).toString.contains("Local Dictionary Exclude")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("st"))
    +    }
    +    descFormatted3.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("dcity"))
    +    }
    +  }
    +
    +  test("test alter set for invalid complex type as include/exclude") {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int,st struct<s_id:int,
    +        | s_name:int,s_city:array<int>>, dcity array<int>)
    +        | STORED BY 'org.apache.carbondata.format'
    +        | tblproperties('local_dictionary_exclude'='name',
    +        | 'local_dictionary_enable'='true')
    +      """.
    +        stripMargin)
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_exclude'='dcity')")
    +    }
    +    intercept[Exception] {
    +      sql("alter table local1 set tblproperties('local_dictionary_include'='st')")
    +    }
    +  }
    +
    +  test("test alter unset for local dictionary disable")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int,add string)
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_enable'='false')
    +      """.stripMargin)
    +
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Enable")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("false"))
    +    }
    +    sql("alter table local1 unset tblproperties('local_dictionary_enable')")
    +    val descLoc2 = sql("describe formatted local1").collect
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Enable")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("true"))
    +    }
    +  }
    +
    +  test("test alter unset for local dictionary enable local dict include")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int,add string)
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_include'='city')
    +      """.stripMargin)
    +
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city"))
    +    }
    +    sql("alter table local1 unset tblproperties('local_dictionary_include')")
    +    val descLoc2 = sql("describe formatted local1").collect
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("name,city"))
    +    }
    +  }
    +
    +  test("test alter unset for local dictionary enable local dict exclude")
    +  {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int,add string)
    +        | STORED BY 'carbondata' tblproperties('local_dictionary_include'='city',
    +        | 'local_dictionary_exclude'='name')
    +      """.stripMargin)
    +
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city"))
    +    }
    +    sql("alter table local1 unset tblproperties('local_dictionary_include')")
    +    val descLoc2 = sql("describe formatted local1").collect
    +    descLoc2.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city"))
    +    }
    +  }
    +
    +  test("test alter unset for valid/invalid complex type as include/exclude") {
    +    sql("drop table if exists local1")
    +    sql(
    +      """
    +        | CREATE TABLE local1(id int, name string, city string, age int,st struct<s_id:string,
    +        | s_name:int,s_city:array<int>>, dcity array<string>)
    +        | STORED BY 'org.apache.carbondata.format'
    +        | tblproperties('local_dictionary_exclude'='st','local_dictionary_include'='city',
    +        | 'local_dictionary_enable'='true')
    +      """.
    +        stripMargin)
    +    val descLoc1 = sql("describe formatted local1").collect
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Include")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("city"))
    +    }
    +    descLoc1.find(_.get(0).toString.contains("Local Dictionary Exclude")) match {
    +      case Some(row) => assert(row.get(1).toString.contains("st"))
    --- End diff --
   
    add check for child column also


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2422
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5484/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2422
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6657/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2422
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5519/



---

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2422#discussion_r199401982
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchemaCommon.scala ---
    @@ -278,14 +282,14 @@ class AlterTableColumnSchemaGenerator(
         // Its based on the dimension name and measure name
         allColumns.filter(x => !x.isInvisible).groupBy(_.getColumnName)
           .foreach(f => if (f._2.size > 1) {
    -      val name = f._1
    -      LOGGER.error(s"Duplicate column found with name: $name")
    -      LOGGER.audit(
    -        s"Validation failed for Create/Alter Table Operation " +
    -        s"for ${ dbName }.${ alterTableModel.tableName }. " +
    -        s"Duplicate column found with name: $name")
    -      sys.error(s"Duplicate column found with name: $name")
    -    })
    +        val name = f._1
    --- End diff --
   
    correct the style


---

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2422#discussion_r199402754
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchemaCommon.scala ---
    @@ -440,7 +544,7 @@ class TableNewProcessor(cm: TableModel) {
       }
     
       // varchar column is a string column that in long_string_columns
    -  private def isVarcharColumn(colName : String): Boolean = {
    --- End diff --
   
    same as above, please crosscheck and remove unnnecessary changes


---

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2422#discussion_r199402302
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchemaCommon.scala ---
    @@ -299,17 +303,112 @@ class AlterTableColumnSchemaGenerator(
         val columnValidator = CarbonSparkFactory.getCarbonColumnValidator
         columnValidator.validateColumns(allColumns)
     
    +
    +    def getLocalDictColumnList(tableProperties: scala.collection.mutable.Map[String, String],
    +        columns: scala.collection.mutable.ListBuffer[ColumnSchema]): (scala.collection.mutable
    +    .ListBuffer[ColumnSchema], scala.collection.mutable.ListBuffer[ColumnSchema]) = {
    +      val includeColumns = new scala.collection.mutable.ListBuffer[ColumnSchema]
    +      val excludeColumns = new scala.collection.mutable.ListBuffer[ColumnSchema]
    +      val localDictIncludeColumns = if (tableProperties
    +        .get(CarbonCommonConstants.LOCAL_DICTIONARY_INCLUDE).isDefined) {
    +        tableProperties(CarbonCommonConstants.LOCAL_DICTIONARY_INCLUDE)
    +      } else {
    +        null
    +      }
    +      val localDictExcludeColumns = if (tableProperties
    +        .get(CarbonCommonConstants.LOCAL_DICTIONARY_EXCLUDE).isDefined) {
    +        tableProperties(CarbonCommonConstants.LOCAL_DICTIONARY_EXCLUDE)
    +      } else {
    +        null
    +      }
    +      if (null != localDictIncludeColumns) {
    +        if (null == localDictExcludeColumns) {
    +          columns.foreach { column =>
    +            if (localDictIncludeColumns.contains(column.getColumnName)) {
    +              includeColumns.append(column)
    +            } else {
    +              if (column.getDataType.equals(DataTypes.STRING) ||
    +                  column.getDataType.toString.equals("ARRAY") ||
    +                  column.getDataType.toString.equals("STRUCT")) {
    +                excludeColumns.append(column)
    +              }
    +            }
    +          }
    +        } else {
    +          columns.foreach { column =>
    +            if (localDictIncludeColumns.contains(column.getColumnName) &&
    +                !localDictExcludeColumns.contains(column.getColumnName)) {
    +              includeColumns.append(column)
    +            } else if (localDictExcludeColumns.contains(column.getColumnName)) {
    +              excludeColumns.append(column)
    +            }
    +          }
    +        }
    +      } else {
    +        if (null == localDictExcludeColumns) {
    +          columns.foreach { column =>
    +            if (column.getDataType.equals(DataTypes.STRING) ||
    +                column.getDataType.toString.equals("ARRAY") ||
    +                column.getDataType.toString.equals("STRUCT")) {
    +              includeColumns.append(column)
    +            }
    +          }
    +        } else {
    +          columns.foreach { column =>
    +            if (!localDictExcludeColumns.contains(column.getColumnName) &&
    +                (column.getDataType.equals(DataTypes.STRING) ||
    +                 column.getDataType.toString.equals("ARRAY") ||
    +                 column.getDataType.toString.equals("STRUCT"))) {
    +              includeColumns.append(column)
    +            } else if (localDictExcludeColumns.contains(column.getColumnName)) {
    +              excludeColumns.append(column)
    +            }
    +          }
    +        }
    +      }
    +
    +      (includeColumns, excludeColumns)
    +    }
    +
    +    val columnsWithoutNewCols = new scala.collection.mutable.ListBuffer[ColumnSchema]
    +    allColumns.foreach { column =>
    +      if (!newCols.exists(x => x.getColumnName.equalsIgnoreCase(column.getColumnName))) {
    +        columnsWithoutNewCols += column
    +      }
    +    }
    +
    +
         if (alterTableModel.tableProperties != null) {
           CarbonUtil
             .setLocalDictColumnsToWrapperSchema(newCols.asJava,
               alterTableModel.tableProperties.asJava,
               tableSchema.getTableProperties.get(CarbonCommonConstants.LOCAL_DICTIONARY_ENABLE))
         }
     
    +    val newMainList = getLocalDictColumnList(tableSchema.getTableProperties.asScala,
    --- End diff --
   
    give proper variable names, add comments in all places


---

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2422#discussion_r199402356
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchemaCommon.scala ---
    @@ -341,7 +445,7 @@ class AlterTableColumnSchemaGenerator(
               }
             }
             else if (elem._1.equalsIgnoreCase("no_inverted_index") &&
    -          (elem._2.split(",").contains(col.getColumnName))) {
    +                 (elem._2.split(",").contains(col.getColumnName))) {
    --- End diff --
   
    remove unnecessary changes


---

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2422#discussion_r199402608
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchemaCommon.scala ---
    @@ -391,7 +495,7 @@ object TableNewProcessor {
         if (dataType == DataTypes.DATE) {
           encoders.add(Encoding.DIRECT_DICTIONARY)
         }
    -    if (dataType == DataTypes.TIMESTAMP && ! highCardinalityDims.contains(colName)) {
    --- End diff --
   
    remove this


---