[GitHub] carbondata pull request #2980: [CARBONDATA-3017] Map DDL Support

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2980
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1706/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2980
 
    Build Failed  with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9966/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2980
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1917/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2980
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1712/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2980
 
    Build Failed  with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9972/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2980
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1923/



---

[qiuchenjian-2]

Github user manishgupta88 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2980#discussion_r240886665
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/model/CarbonLoadModel.java ---
    @@ -65,8 +65,7 @@
       private String csvHeader;
       private String[] csvHeaderColumns;
       private String csvDelimiter;
    -  private String complexDelimiterLevel1;
    -  private String complexDelimiterLevel2;
    +  private ArrayList<String> complexDelimiters = new ArrayList<>();
    --- End diff --
   
    We can do lazy initialization in the setter method. This will avoid extra memory being consumed for non complex type schemas


---

[qiuchenjian-2]

Github user manishgupta88 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2980#discussion_r240886853
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/model/CarbonLoadModel.java ---
    @@ -631,7 +620,9 @@ public void setFactTimeStamp(long factTimeStamp) {
       }
     
       public String[] getDelimiters() {
    -    return new String[] { complexDelimiterLevel1, complexDelimiterLevel2 };
    +    String[] delimiters = new String[complexDelimiters.size()];
    +    delimiters = complexDelimiters.toArray(delimiters);
    +    return delimiters;
    --- End diff --
   
    This method is not required. Conversion can be done whereever required


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2980
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1716/



---

[qiuchenjian-2]

Github user manishnalla1994 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2980#discussion_r240900542
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/model/CarbonLoadModel.java ---
    @@ -65,8 +65,7 @@
       private String csvHeader;
       private String[] csvHeaderColumns;
       private String csvDelimiter;
    -  private String complexDelimiterLevel1;
    -  private String complexDelimiterLevel2;
    +  private ArrayList<String> complexDelimiters = new ArrayList<>();
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user manishnalla1994 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2980#discussion_r240900568
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/parser/impl/RowParserImpl.java ---
    @@ -34,8 +37,12 @@
       private int numberOfColumns;
     
       public RowParserImpl(DataField[] output, CarbonDataLoadConfiguration configuration) {
    -    String[] complexDelimiters =
    +    String[] tempComplexDelimiters =
             (String[]) configuration.getDataLoadProperty(DataLoadProcessorConstants.COMPLEX_DELIMITERS);
    +    Queue<String> complexDelimiters = new LinkedList<>();
    +    for (int i = 0; i < 4; i++) {
    --- End diff --
   
    done



---

[qiuchenjian-2]

Github user manishnalla1994 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2980#discussion_r240900612
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/parser/CarbonParserFactory.java ---
    @@ -51,23 +54,37 @@ public static GenericParser createParser(CarbonColumn carbonColumn, String[] com
        *                           delimiters
        * @return GenericParser
        */
    -  private static GenericParser createParser(CarbonColumn carbonColumn, String[] complexDelimiters,
    +  private static GenericParser createParser(CarbonColumn carbonColumn,
    +      Queue<String> complexDelimiters,
           String nullFormat, int depth) {
    +    if (depth > 2) {
    +      return null;
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user manishnalla1994 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2980#discussion_r240900628
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/model/LoadOption.java ---
    @@ -119,6 +119,10 @@
             "complex_delimiter_level_2",
             Maps.getOrDefault(options, "complex_delimiter_level_2", ":"));
     
    +    optionsFinal.put(
    +        "complex_delimiter_level_3",
    +        Maps.getOrDefault(options, "complex_delimiter_level_3", "\\\\003"));
    +
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user manishnalla1994 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2980#discussion_r240900666
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/model/CarbonLoadModel.java ---
    @@ -631,7 +651,7 @@ public void setFactTimeStamp(long factTimeStamp) {
       }
     
       public String[] getDelimiters() {
    -    return new String[] { complexDelimiterLevel1, complexDelimiterLevel2 };
    +    return new String[] { complexDelimiterLevel1, complexDelimiterLevel2, complexDelimiterLevel3 };
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user manishnalla1994 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2980#discussion_r240900688
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/DataLoadProcessBuilder.java ---
    @@ -222,8 +222,8 @@ public static CarbonDataLoadConfiguration createConfiguration(CarbonLoadModel lo
         configuration.setSegmentId(loadModel.getSegmentId());
         configuration.setTaskNo(loadModel.getTaskNo());
         configuration.setDataLoadProperty(DataLoadProcessorConstants.COMPLEX_DELIMITERS,
    -        new String[] { loadModel.getComplexDelimiterLevel1(),
    -            loadModel.getComplexDelimiterLevel2() });
    +        new String[] { loadModel.getComplexDelimiterLevel1(), loadModel.getComplexDelimiterLevel2(),
    +            loadModel.getComplexDelimiterLevel3(), loadModel.getComplexDelimiterLevel4() });
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user manishnalla1994 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2980#discussion_r240900706
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala ---
    @@ -188,11 +188,13 @@ case class CarbonLoadDataCommand(
         val carbonLoadModel = new CarbonLoadModel()
         val tableProperties = table.getTableInfo.getFactTable.getTableProperties
         val optionsFinal = LoadOption.fillOptionWithDefaultValue(options.asJava)
    +    // These two delimiters are non configurable and hardcoded for map type
    +    optionsFinal.put("complex_delimiter_level_3", "\003")
    +    optionsFinal.put("complex_delimiter_level_4", "\004")
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user manishnalla1994 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2980#discussion_r240900720
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateDDLForComplexMapType.scala ---
    @@ -0,0 +1,452 @@
    +/*
    +
    +    Licensed to the Apache Software Foundation (ASF) under one or more
    +    contributor license agreements. See the NOTICE file distributed with
    +    this work for additional information regarding copyright ownership.
    +    The ASF licenses this file to You under the Apache License, Version 2.0
    +    (the "License"); you may not use this file except in compliance with
    +    the License. You may obtain a copy of the License at
    +    *
    +    http://www.apache.org/licenses/LICENSE-2.0
    +    *
    +    Unless required by applicable law or agreed to in writing, software
    +    distributed under the License is distributed on an "AS IS" BASIS,
    +    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +    See the License for the specific language governing permissions and
    +    limitations under the License.
    +    */
    +package org.apache.carbondata.spark.testsuite.createTable.TestCreateDDLForComplexMapType
    +
    +import java.io.File
    +import java.util
    +
    +import org.apache.hadoop.conf.Configuration
    +import org.apache.spark.sql.{AnalysisException, Row}
    +import org.apache.spark.sql.test.util.QueryTest
    +import org.scalatest.BeforeAndAfterAll
    +
    +import org.apache.carbondata.core.datastore.chunk.impl.DimensionRawColumnChunk
    +
    +class TestCreateDDLForComplexMapType extends QueryTest with BeforeAndAfterAll {
    +  private val conf: Configuration = new Configuration(false)
    +
    +  val rootPath = new File(this.getClass.getResource("/").getPath
    +                          + "../../../..").getCanonicalPath
    +
    +  val path = s"$rootPath/examples/spark2/src/main/resources/maptest2.csv"
    +
    +  private def checkForLocalDictionary(dimensionRawColumnChunks: util
    +  .List[DimensionRawColumnChunk]): Boolean = {
    +    var isLocalDictionaryGenerated = false
    +    import scala.collection.JavaConversions._
    +    for (dimensionRawColumnChunk <- dimensionRawColumnChunks) {
    +      if (dimensionRawColumnChunk.getDataChunkV3
    +        .isSetLocal_dictionary) {
    +        isLocalDictionaryGenerated = true
    +      }
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user manishnalla1994 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2980#discussion_r240900728
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java ---
    @@ -338,12 +338,15 @@ public static CarbonLoadModel getLoadModel(Configuration conf) throws IOExceptio
                 SKIP_EMPTY_LINE,
                 carbonProperty.getProperty(CarbonLoadOptionConstants.CARBON_OPTIONS_SKIP_EMPTY_LINE)));
     
    -    String complexDelim = conf.get(COMPLEX_DELIMITERS, "$" + "," + ":");
    +    String complexDelim = conf.get(COMPLEX_DELIMITERS, "$" + "," + ":" + "," + "\\\\003");
         String[] split = complexDelim.split(",");
         model.setComplexDelimiterLevel1(split[0]);
         if (split.length > 1) {
           model.setComplexDelimiterLevel2(split[1]);
         }
    +    if (split.length > 2) {
    +      model.setComplexDelimiterLevel3(split[2]);
    +    }
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user manishnalla1994 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2980#discussion_r240900751
 
    --- Diff: examples/spark2/src/main/resources/maptest2.csv ---
    @@ -0,0 +1,2 @@
    +1\002Nalla\0012\002Singh\0011\002Gupta\0014\002Kumar
    +10\002Nallaa\00120\002Sissngh\001100\002Gusspta\00140\002Kumar
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user manishnalla1994 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2980#discussion_r240900767
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java ---
    @@ -338,12 +338,15 @@ public static CarbonLoadModel getLoadModel(Configuration conf) throws IOExceptio
                 SKIP_EMPTY_LINE,
                 carbonProperty.getProperty(CarbonLoadOptionConstants.CARBON_OPTIONS_SKIP_EMPTY_LINE)));
     
    -    String complexDelim = conf.get(COMPLEX_DELIMITERS, "$" + "," + ":");
    +    String complexDelim = conf.get(COMPLEX_DELIMITERS, "$" + "," + ":" + "," + "\\\\003");
    --- End diff --
   
    done


---