Babulal created CARBONDATA-404:
----------------------------------
Summary: Data loading from DataFrame to carbon table is FAILED
Key: CARBONDATA-404
URL:
https://issues.apache.org/jira/browse/CARBONDATA-404 Project: CarbonData
Issue Type: Bug
Components: data-load
Affects Versions: 0.1.0-incubating
Reporter: Babulal
Data loading FAILED when Loading data from DataFrame with tempCSV option =true (Default option ) in 3 Node cluster .
Steps
val customSchema = StructType(Array( StructField("imei", StringType, true), StructField("deviceInformationId", IntegerType, true), StructField("mac", StringType, true), StructField("productdate", TimestampType , true), StructField("updatetime", TimestampType, true), StructField("gamePointId", DoubleType, true), StructField("contractNumber", DoubleType, true) ));
val df = cc.read.format("com.databricks.spark.csv").option("header", "false").schema(customSchema).load("/opt/data/xyz/100_default_date_11_header.csv");
Start data loading
scala> df.write.format("carbondata").option("tableName","mycarbon2").save();
INFO 10-11 23:24:35,970 - main Query [
CREATE TABLE IF NOT EXISTS DEFAULT.MYCARBON2
(IMEI STRING, DEVICEINFORMATIONID INT, MAC STRING, PRODUCTDATE TIMESTAMP, UPDATETIME TIMESTAMP, GAMEPOINTID DOUBLE, CONTRACTNUMBER DOUBLE)
STORED BY 'ORG.APACHE.CARBONDATA.FORMAT'
]
INFO 10-11 23:24:35,977 - Parsing command:
CREATE TABLE IF NOT EXISTS default.mycarbon2
(imei STRING, deviceInformationId INT, mac STRING, productdate TIMESTAMP, updatetime TIMESTAMP, gamePointId DOUBLE, contractNumber DOUBLE)
STORED BY 'org.apache.carbondata.format'
INFO 10-11 23:24:35,978 - Parse Completed
INFO 10-11 23:24:36,227 - main Query [
LOAD DATA INPATH './TEMPCSV'
INTO TABLE DEFAULT.MYCARBON2
OPTIONS ('FILEHEADER' = 'IMEI,DEVICEINFORMATIONID,MAC,PRODUCTDATE,UPDATETIME,GAMEPOINTID,CONTRACTNUMBER')
]
INFO 10-11 23:24:36,233 - Successfully able to get the table metadata file lock
AUDIT 10-11 23:24:36,234 - [BLR1000007781][root][Thread-1]Dataload failed for default.mycarbon2. The input file does not exist: ./tempCSV
INFO 10-11 23:24:36,234 - main Successfully deleted the lock file /tmp/default/mycarbon2/meta.lock
INFO 10-11 23:24:36,234 - Table MetaData Unlocked Successfully after data load
org.apache.carbondata.processing.etl.DataLoadingException: The input file does not exist: ./tempCSV
at org.apache.spark.util.FileUtils$$anonfun$getPaths$1.apply$mcVI$sp(FileUtils.scala:66)
CSV DATA
1AA1,1,Mikaa1,2015-01-01 11:00:00,2015-01-01 13:00:00,198,260
1AA2,3,Mikaa2,2015-01-02 12:00:00,2015-01-01 14:00:00,278,230
1AA3,1,Mikaa1,2015-01-03 13:00:00,2015-01-01 15:00:00,2556,1
1AA4,10,Mikaa2,2015-01-04 14:00:00,2015-01-01 16:00:00,640,254
1AA5,10,Mikaa,2015-01-05 15:00:00,2015-01-01 17:00:00,980,256
1AA6,10,Mikaa,2015-01-06 16:00:00,2015-01-01 18:00:00,1,2378
1AA7,10,Mikaa,2015-01-07 17:00:00,2015-01-01 19:00:00,96,234
1AA8,9,max,2015-01-08 18:00:00,2015-01-01 20:00:00,89,236
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Locked
