read parquet file --No such file or directory

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

read parquet file --No such file or directory

lionel061201
Hi,
I got a strange error that carbonContext can not read parquet file on hdfs
in my program. I test on spark shell and it returns the same error,
anything goes wrong?

>>>>>>Test for sqlContext, Success
----------------------------------------------------------------------------------------------------------------

Spark context available as sc (master = yarn-client, app id =
application_1479961381214_0551).

SQL context available as sqlContext.



scala> val parquetFile = sqlContext.read.parquet("/
user/hive/default/testdata_parquet_all")

parquetFile: org.apache.spark.sql.DataFrame = [id: double,***************...





>>>>>>Test for CarbonContext, Failed

------------------------------------------------------------------------------------------------------------------

Spark context available as sc (master = yarn-client, app id =
application_1479961381214_0552).Hi

SQL context available as sqlContext.



scala> import org.apache.spark.sql.CarbonContext

import org.apache.spark.sql.CarbonContext



scala> val cc = new CarbonContext(sc)

cc: org.apache.spark.sql.CarbonContext = org.apache.spark.sql.
CarbonContext@3574122f



scala> val parquetFile = cc.read.parquet("/user/hive/
default/testdata_parquet_all")

AUDIT 30-11 13:42:24,114 - [*******][appuser][Thread-1]Creating timestamp
file for .

java.io.IOException: No such file or directory

         at java.io.UnixFileSystem.createFileExclusively(Native Method)

         at java.io.File.createNewFile(File.java:1006)

         at org.apache.carbondata.core.datastorage.store.impl.
FileFactory.createNewFile(FileFactory.java:372)

         at org.apache.spark.sql.hive.CarbonMetastoreCatalog.
updateSchemasUpdatedTime(CarbonMetastoreCatalog.scala:468)

         at org.apache.spark.sql.hive.CarbonMetastoreCatalog.loadMetadata(
CarbonMetastoreCatalog.scala:181)

         at org.apache.spark.sql.hive.CarbonMetastoreCatalog.<init>(
CarbonMetastoreCatalog.scala:114)
Reply | Threaded
Open this post in threaded view
|

Re: read parquet file --No such file or directory

ZhuWilliam
Can you try to create CarbonContext  as follows:

val cc = new CarbonContext(sc, storeLocation, metaLocation")

cc.setConf("carbon.kettle.home", kettleHome)