[jira] [Created] (CARBONDATA-3161) Pipe "|" dilimiter is not working for streaming table

[Akash R Nilugal (Jira)]

Pawan Malwal created CARBONDATA-3161:
----------------------------------------

             Summary: Pipe "|" dilimiter is not working for streaming table
                 Key: CARBONDATA-3161
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-3161
             Project: CarbonData
          Issue Type: Bug
          Components: data-load
            Reporter: Pawan Malwal
            Assignee: Pawan Malwal


csv data with "|" as a dilimiter is not getting loaded into streaming table correctly.

*DDL:*

create table table1_st(begintime TIMESTAMP, deviceid STRING, statcycle INT, topologypath STRING, devicetype STRING, rebootnum INT) stored by 'carbondata' TBLPROPERTIES('SORT_SCOPE'='GLOBAL_SORT','sort_columns'='deviceid,begintime','streaming' ='true');

*Run in spark shell:*

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.SparkSession.Builder;
import org.apache.spark.sql.CarbonSession;
import org.apache.spark.sql.CarbonSession.CarbonBuilder;
import org.apache.spark.sql.streaming._
import org.apache.carbondata.streaming.parser._


val enableHiveSupport = SparkSession.builder().enableHiveSupport();
val carbon=new CarbonBuilder(enableHiveSupport).getOrCreateCarbonSession("hdfs://hacluster/user/hive/warehouse/")
val df=carbon.readStream.text("/user/*.csv")

val qrymm_0001 = df.writeStream.format("carbondata").option(CarbonStreamParser.CARBON_STREAM_PARSER, CarbonStreamParser.CARBON_STREAM_PARSER_CSV).{color:#FF0000}*option("delimiter","|")*{color}.option("header","false").option("dbName","stdb").option("checkpointLocation", "/tmp/tb1").option("bad_records_action","FORCE").option("tableName","table1_st").trigger(ProcessingTime(6000)).option("carbon.streaming.auto.handoff.enabled","true").option("TIMESTAMPFORMAT","yyyy-dd-MM HH:mm:ss").start

 

*Sample records:*
 begintime| deviceid| statcycle| topologypath| devicetype| rebootnum
 2018-10-01 00:00:00|Device1|0|dsad|STB|9
 2018-10-01 00:05:00|Device1|0|Rsad|STB|4
 2018-10-01 00:10:00|Device1|0|fsf|STB|6
 2018-10-01 00:15:00|Device1|0|fdgf|STB|8



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)