anubhav tarar created CARBONDATA-1990:
-----------------------------------------
Summary: Null values shown when the basic word count example is tried on carbon streaming table
Key: CARBONDATA-1990
URL:
https://issues.apache.org/jira/browse/CARBONDATA-1990 Project: CarbonData
Issue Type: Bug
Components: spark-integration
Affects Versions: 1.3.0
Environment: spark-2.1
Reporter: anubhav tarar
Fix For: 1.3.0
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.CarbonSession._
import org.apache.spark.sql.types._
import org.apache.spark.sql.{AnalysisException, DataFrame, DataFrameWriter, Row, SaveMode}
val carbon = SparkSession.builder().config(sc.getConf) .getOrCreateCarbonSession("hdfs://localhost:54311/newCarbonStore","/tmp")
carbon.sql("drop table if exists stream_table_csvSource")
carbon.sql("create table stream_table_csvSource(word string,count string) stored by 'carbondata' TBLPROPERTIES('streaming'='true')")
import carbon.sqlContext.implicits._
val userSchema = StructType(
Array(StructField("name", StringType)))
val lines = carbon.readStream
.format("csv")
.option("path", "file:///home/anubhav/Documents/csv")
.option("sep",",")
.schema(userSchema)
.load()
lines.printSchema
val words = lines.as[String].flatMap(_.split("\n"))
val wordCounts = words.groupBy("value").count()
val qry = wordCounts.writeStream
.outputMode("complete")
.format("carbondata")
.option("checkpointLocation","file/home/anubhav/Downloads/checkpointlist155")
.option("dbName", "default")
.option("tableName", "stream_table_csvSource")
.start()
qry.awaitTermination()
then i close my shell and select data from table there are null values shown
scala> carbon.sql("select * from stream_table_csvSource").show
18/01/05 14:14:34 ERROR CarbonProperties: Executor task launch worker-0 Configured value for property carbon.number.of.cores.while.loading is wrong. Falling back to the default value 2
+-------+-----+
| word|count|
+-------+-----+
|anubhav| null|
|geetika| null|
| prince| null|
+-------+-----+
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Locked
