[GitHub] carbondata pull request #2695: [CARBONDATA-2919] Support ingest from Kafka i...

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/164/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/332/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8403/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8435/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/197/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/365/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2695#discussion_r216306167
 
    --- Diff: pom.xml ---
    @@ -217,6 +217,12 @@
             <version>${spark.version}</version>
             <scope>${spark.deps.scope}</scope>
           </dependency>
    +      <dependency>
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2695#discussion_r216309957
 
    --- Diff: streaming/src/main/java/org/apache/carbondata/streaming/CarbonStreamRecordWriter.java ---
    @@ -316,6 +316,10 @@ private void appendBlockletToDataFile() throws IOException {
       }
     
       public BlockletMinMaxIndex getBatchMinMaxIndex() {
    +    if (output == null) {
    +      return StreamSegment.mergeBlockletMinMax(
    --- End diff --
   
    fixed


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2695#discussion_r216310510
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java ---
    @@ -1182,6 +1182,15 @@ private static void setLocalDictInfo(CarbonTable table, TableInfo tableInfo) {
         }
       }
     
    +  /**
    +   * Return the format value defined in table properties
    +   * @return String as per table properties, null if not defined
    +   */
    +  public String getFormat() {
    +    return getTableInfo().getFactTable().getTableProperties()
    +            .get("format");
    --- End diff --
   
    fixed


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2695#discussion_r216310962
 
    --- Diff: integration/spark2/src/main/scala/org/apache/carbondata/stream/StreamJobManager.scala ---
    @@ -52,19 +52,24 @@ object StreamJobManager {
         }
       }
     
    -  private def validateSinkTable(querySchema: StructType, sink: CarbonTable): Unit = {
    +  private def validateSinkTable(validateQuerySchema: Boolean,
    +                                querySchema: StructType, sink: CarbonTable): Unit = {
         if (!sink.isStreamingSink) {
           throw new MalformedCarbonCommandException(s"Table ${sink.getTableName} is not " +
                                                     "streaming sink table " +
                                                     "('streaming' tblproperty is not 'sink' or 'true')")
         }
    -    val fields = sink.getCreateOrderColumn(sink.getTableName).asScala.map { column =>
    -      StructField(column.getColName,
    -        CarbonSparkDataSourceUtil.convertCarbonToSparkDataType(column.getDataType))
    -    }
    -    if (!querySchema.equals(StructType(fields))) {
    -      throw new MalformedCarbonCommandException(s"Schema of table ${sink.getTableName} " +
    -                                                s"does not match query output")
    +    // TODO: validate query schema against sink in kafka (we cannot get schema directly)
    +    if (validateQuerySchema) {
    +      val fields = sink.getCreateOrderColumn(sink.getTableName).asScala.map { column =>
    +        StructField(
    +          column.getColName,
    +          CarbonSparkDataSourceUtil.convertCarbonToSparkDataType(column.getDataType))
    +      }
    +      if (!querySchema.equals(StructType(fields))) {
    +        throw new MalformedCarbonCommandException(s"Schema of table ${ sink.getTableName } " +
    --- End diff --
   
    fixed


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/371/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8441/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/203/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8442/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/372/



---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/214/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/383/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8453/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/215/



---