[GitHub] carbondata pull request #2401: [CARBONDATA-2585][CARBONDATA-2586][Local Dict...

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2401#discussion_r198441455
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
    @@ -3041,24 +3041,30 @@ public static String getBlockId(AbsoluteTableIdentifier identifier, String fileP
        * @param mainTableProperties
        */
       public static void setLocalDictColumnsToWrapperSchema(List<ColumnSchema> columns,
    -      Map<String, String> mainTableProperties) {
    -    String isLocalDictEnabledForMainTable =
    -        mainTableProperties.get(CarbonCommonConstants.LOCAL_DICTIONARY_ENABLE);
    -    String localDictIncludeColumnsOfMainTable =
    -        mainTableProperties.get(CarbonCommonConstants.LOCAL_DICTIONARY_INCLUDE);
    -    String localDictExcludeColumnsOfMainTable =
    -        mainTableProperties.get(CarbonCommonConstants.LOCAL_DICTIONARY_EXCLUDE);
    +      Map<String, String> mainTableProperties, String isLocalDictEnabledForMainTable) {
         String[] listOfDictionaryIncludeColumns = null;
         String[] listOfDictionaryExcludeColumns = null;
    +    String localDictIncludeColumns =
    +        mainTableProperties.get(CarbonCommonConstants.LOCAL_DICTIONARY_INCLUDE);
    +    String localDictExcludeColumns =
    +        mainTableProperties.get(CarbonCommonConstants.LOCAL_DICTIONARY_EXCLUDE);
    +    if (null != localDictIncludeColumns) {
    +      listOfDictionaryIncludeColumns = localDictIncludeColumns.trim().split("\\s*,\\s*");
    --- End diff --
   
    if user gives comma seperated values with empty spaces, need to trim it and then check validate the column


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2401#discussion_r198442294
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala ---
    @@ -615,4 +618,114 @@ object CarbonScalaUtil {
           sparkSession,
           schema)._1.asInstanceOf[Object]
       }
    +
    +  /**
    +   * this method validates the local dictionary columns configurations
    +   *
    +   * @param tableProperties
    +   * @param localDictColumns
    +   */
    +  def validateLocalDictionaryColumns(tableProperties: mutable.Map[String, String],
    +      localDictColumns: Seq[String]): Unit = {
    +    var dictIncludeColumns: Seq[String] = Seq[String]()
    +
    +    // check if the duplicate columns are specified in table schema
    +    if (localDictColumns.distinct.lengthCompare(localDictColumns.size) != 0) {
    +      val duplicateColumns = (dictIncludeColumns ++ localDictColumns)
    +        .diff((dictIncludeColumns ++ localDictColumns).distinct).distinct
    +      val errMsg =
    +        "LOCAL_DICTIONARY_INCLUDE/LOCAL_DICTIONARY_EXCLUDE contains Duplicate Columns: " +
    +        duplicateColumns.mkString(",") +
    +        ". Please check create table statement."
    +      throw new MalformedCarbonCommandException(errMsg)
    +    }
    +
    +    // check if the same column is present in both dictionary include and local dictionary columns
    +    // configuration
    +    if (tableProperties.get(CarbonCommonConstants.DICTIONARY_INCLUDE).isDefined) {
    +      dictIncludeColumns =
    +        tableProperties(CarbonCommonConstants.DICTIONARY_INCLUDE).split(",").map(_.trim)
    +      localDictColumns.foreach { distCol =>
    +        if (dictIncludeColumns.exists(x => x.equalsIgnoreCase(distCol.trim))) {
    +          val duplicateColumns = (dictIncludeColumns ++ localDictColumns)
    +            .diff((dictIncludeColumns ++ localDictColumns).distinct).distinct
    --- End diff --
   
    I think you require common columns not difference columns here


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6580/



---

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2401#discussion_r198443241
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala ---
    @@ -615,4 +618,114 @@ object CarbonScalaUtil {
           sparkSession,
           schema)._1.asInstanceOf[Object]
       }
    +
    +  /**
    +   * this method validates the local dictionary columns configurations
    +   *
    +   * @param tableProperties
    +   * @param localDictColumns
    +   */
    +  def validateLocalDictionaryColumns(tableProperties: mutable.Map[String, String],
    +      localDictColumns: Seq[String]): Unit = {
    +    var dictIncludeColumns: Seq[String] = Seq[String]()
    +
    +    // check if the duplicate columns are specified in table schema
    +    if (localDictColumns.distinct.lengthCompare(localDictColumns.size) != 0) {
    +      val duplicateColumns = (dictIncludeColumns ++ localDictColumns)
    --- End diff --
   
    yes, it is not required, will remove that


---

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2401#discussion_r198443925
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala ---
    @@ -615,4 +618,114 @@ object CarbonScalaUtil {
           sparkSession,
           schema)._1.asInstanceOf[Object]
       }
    +
    +  /**
    +   * this method validates the local dictionary columns configurations
    +   *
    +   * @param tableProperties
    +   * @param localDictColumns
    +   */
    +  def validateLocalDictionaryColumns(tableProperties: mutable.Map[String, String],
    +      localDictColumns: Seq[String]): Unit = {
    +    var dictIncludeColumns: Seq[String] = Seq[String]()
    +
    +    // check if the duplicate columns are specified in table schema
    +    if (localDictColumns.distinct.lengthCompare(localDictColumns.size) != 0) {
    +      val duplicateColumns = (dictIncludeColumns ++ localDictColumns)
    +        .diff((dictIncludeColumns ++ localDictColumns).distinct).distinct
    +      val errMsg =
    +        "LOCAL_DICTIONARY_INCLUDE/LOCAL_DICTIONARY_EXCLUDE contains Duplicate Columns: " +
    +        duplicateColumns.mkString(",") +
    +        ". Please check create table statement."
    +      throw new MalformedCarbonCommandException(errMsg)
    +    }
    +
    +    // check if the same column is present in both dictionary include and local dictionary columns
    +    // configuration
    +    if (tableProperties.get(CarbonCommonConstants.DICTIONARY_INCLUDE).isDefined) {
    +      dictIncludeColumns =
    +        tableProperties(CarbonCommonConstants.DICTIONARY_INCLUDE).split(",").map(_.trim)
    +      localDictColumns.foreach { distCol =>
    +        if (dictIncludeColumns.exists(x => x.equalsIgnoreCase(distCol.trim))) {
    +          val duplicateColumns = (dictIncludeColumns ++ localDictColumns)
    +            .diff((dictIncludeColumns ++ localDictColumns).distinct).distinct
    --- End diff --
   
    basically, the variable name should be changed to common column, i will change that, functionally it is getting the common column


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5409/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5477/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6590/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5480/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5483/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6600/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5419/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5429/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6628/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5457/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5504/



---

[qiuchenjian-2]

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6633/



---

[qiuchenjian-2]

Github user akashrn5 commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    retest this please


---

[qiuchenjian-2]

Github user akashrn5 commented on the issue:

    https://github.com/apache/carbondata/pull/2401
 
    retest sdv please


---