Posted by
xm_zzc on
Nov 16, 2018; 3:53am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Throw-NoSuchElementException-None-get-error-when-use-CarbonSession-to-read-parquet-tp68467.html
Hi:
Please help. I used CarbonSession to read parquet and it throws
'NoSuchElementException: None.get' error, reading carbondata files is ok.
*Env*: local mode, Spark 2.3 + CarbonData(master branch)
*Code*:
import org.apache.spark.sql.CarbonSession._
val spark = SparkSession
.builder()
.master("local[1]")
.appName("Carbon1_5")
.config("spark.sql.warehouse.dir", warehouse)
.config("spark.default.parallelism", 4)
.config("spark.sql.shuffle.partitions", 4)
.getOrCreateCarbonSession(storeLocation, Constants.METASTORE_DB)
spark.conf.set("spark.sql.parquet.binaryAsString", true)
val parquets = spark.read.parquet("/data1/parquets/")
println(parquets.count())
*Error*:
Exception in thread "main" java.util.ServiceConfigurationError:
org.apache.spark.sql.sources.DataSourceRegister: Provider
org.apache.spark.sql.carbondata.execution.datasources.SparkCarbonFileFormat
could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$class.foreach(Iterator.scala:742)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:258)
at scala.collection.TraversableLike$class.filter(TraversableLike.scala:270)
at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
at
org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:618)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:190)
at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:622)
at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:606)
at cn.xm.zzc.carbonmaster.Carbon1_5$.testReadSpeed(Carbon1_5.scala:434)
at cn.xm.zzc.carbonmaster.Carbon1_5$.main(Carbon1_5.scala:105)
at cn.xm.zzc.carbonmaster.Carbon1_5.main(Carbon1_5.scala)
Caused by: java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at
org.apache.spark.sql.carbondata.execution.datasources.SparkCarbonFileFormat.<init>(SparkCarbonFileFormat.scala:120)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.lang.Class.newInstance(Class.java:442)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 17 more
Thanks.
--
Sent from:
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/