[GitHub] carbondata pull request #2695: [CARBONDATA-2919] Support ingest from Kafka i...

classic Classic list List threaded Threaded
74 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/384/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8454/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8458/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/219/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/388/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2695: [CARBONDATA-2919] Support ingest from Kafka i...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2695#discussion_r216598273
 
    --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/StreamSQLExample.scala ---
    @@ -0,0 +1,124 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.examples
    +
    +import java.io.File
    +
    +import org.apache.carbondata.examples.util.ExampleUtils
    +
    +// scalastyle:off println
    +object StreamSQLExample {
    --- End diff --
   
    How to run this example? I am getting the following error
    ```
    Exception in thread "main" org.apache.spark.sql.AnalysisException: resolved attribute(s) L_ORDERKEY#188 missing from l_receiptdate#286,l_commitdate#285,l_orderkey#274,l_tax#281,l_returnflag#282,l_linenumber#277,l_discount#280,l_partkey#275,l_shipdate#284,l_shipmode#288,l_linestatus#283,l_shipinstruct#287,l_extendedprice#279,l_comment#289,l_quantity#278,l_suppkey#276 in operator !Filter ((L_ORDERKEY#188 % 2) = 0);;
    Project [l_orderkey#274, l_partkey#275, l_suppkey#276, l_linenumber#277, l_quantity#278, l_extendedprice#279, l_discount#280, l_tax#281, l_returnflag#282, l_linestatus#283, l_shipdate#284, l_commitdate#285, l_receiptdate#286, l_shipinstruct#287, l_shipmode#288, l_comment#289]
    +- !Filter ((L_ORDERKEY#188 % 2) = 0)
       +- SubqueryAlias source
          +- Project [CASE WHEN (size(_values#271) > 0) THEN _values#271[0] ELSE cast(null as string) END AS l_orderkey#274, CASE WHEN (size(_values#271) > 1) THEN _values#271[1] ELSE cast(null as string) END AS l_partkey#275, CASE WHEN (size(_values#271) > 2) THEN _values#271[2] ELSE cast(null as string) END AS l_suppkey#276, CASE WHEN (size(_values#271) > 3) THEN _values#271[3] ELSE cast(null as string) END AS l_linenumber#277, CASE WHEN (size(_values#271) > 4) THEN _values#271[4] ELSE cast(null as string) END AS l_quantity#278, CASE WHEN (size(_values#271) > 5) THEN _values#271[5] ELSE cast(null as string) END AS l_extendedprice#279, CASE WHEN (size(_values#271) > 6) THEN _values#271[6] ELSE cast(null as string) END AS l_discount#280, CASE WHEN (size(_values#271) > 7) THEN _values#271[7] ELSE cast(null as string) END AS l_tax#281, CASE WHEN (size(_values#271) > 8) THEN _values#271[8] ELSE cast(null as string) END AS l_returnflag#282, CASE WHEN (size(_values#271) > 9) THEN _values#
 271[9] ELSE cast(null as string) END AS l_linestatus#283, CASE WHEN (size(_values#271) > 10) THEN _values#271[10] ELSE cast(null as string) END AS l_shipdate#284, CASE WHEN (size(_values#271) > 11) THEN _values#271[11] ELSE cast(null as string) END AS l_commitdate#285, CASE WHEN (size(_values#271) > 12) THEN _values#271[12] ELSE cast(null as string) END AS l_receiptdate#286, CASE WHEN (size(_values#271) > 13) THEN _values#271[13] ELSE cast(null as string) END AS l_shipinstruct#287, CASE WHEN (size(_values#271) > 14) THEN _values#271[14] ELSE cast(null as string) END AS l_shipmode#288, CASE WHEN (size(_values#271) > 15) THEN _values#271[15] ELSE cast(null as string) END AS l_comment#289]
             +- Project [split(_value#268, \|) AS _values#271]
                +- Project [cast(value#254 as string) AS _value#268]
                   +- StreamingRelation DataSource(org.apache.spark.sql.CarbonSession@5e6ad84c,kafka,List(),None,List(),None,Map(format -> kafka, streaming -> source, subscribe -> test, local_dictionary_enable -> false, kafka.bootstrap.servers -> localhost:9092, sort_columns -> l_suppkey,l_returnflag,l_linestatus,l_shipdate,l_commitdate,l_receiptdate,l_shipinstruct,l_shipmode,l_comment, comment -> , bad_records_path -> , delimiter -> |),None), kafka, [key#253, value#254, topic#255, partition#256, offset#257L, timestamp#258, timestampType#259]
    ```


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2695: [CARBONDATA-2919] Support ingest from Kafka i...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2695#discussion_r216736902
 
    --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/StreamSQLExample.scala ---
    @@ -0,0 +1,124 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.examples
    +
    +import java.io.File
    +
    +import org.apache.carbondata.examples.util.ExampleUtils
    +
    +// scalastyle:off println
    +object StreamSQLExample {
    --- End diff --
   
    Now I changed this Example to use socket stream source.


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8477/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/238/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/407/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/241/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8480/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/410/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    retest this please



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/252/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8491/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/421/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8492/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/253/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2695
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/256/



---
1234