Throw 'NoSuchElementException: None.get' error when use CarbonSession to read parquet.

[xm_zzc]

Hi:
  Please help. I used CarbonSession to read parquet and it throws
'NoSuchElementException: None.get' error, reading carbondata files is ok.
  *Env*: local mode, Spark 2.3 + CarbonData(master branch)
  *Code*:
    import org.apache.spark.sql.CarbonSession._
    val spark = SparkSession
      .builder()
      .master("local[1]")
      .appName("Carbon1_5")
      .config("spark.sql.warehouse.dir", warehouse)
      .config("spark.default.parallelism", 4)
      .config("spark.sql.shuffle.partitions", 4)
      .getOrCreateCarbonSession(storeLocation, Constants.METASTORE_DB)
    spark.conf.set("spark.sql.parquet.binaryAsString", true)
    val parquets = spark.read.parquet("/data1/parquets/")
    println(parquets.count())

  *Error*:
    Exception in thread "main" java.util.ServiceConfigurationError:
org.apache.spark.sql.sources.DataSourceRegister: Provider
org.apache.spark.sql.carbondata.execution.datasources.SparkCarbonFileFormat
could not be instantiated
        at java.util.ServiceLoader.fail(ServiceLoader.java:232)
        at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
        at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
        at scala.collection.Iterator$class.foreach(Iterator.scala:742)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at
scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:258)
        at scala.collection.TraversableLike$class.filter(TraversableLike.scala:270)
        at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
        at
org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:618)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:190)
        at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:622)
        at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:606)
        at cn.xm.zzc.carbonmaster.Carbon1_5$.testReadSpeed(Carbon1_5.scala:434)
        at cn.xm.zzc.carbonmaster.Carbon1_5$.main(Carbon1_5.scala:105)
        at cn.xm.zzc.carbonmaster.Carbon1_5.main(Carbon1_5.scala)
Caused by: java.util.NoSuchElementException: None.get
        at scala.None$.get(Option.scala:347)
        at scala.None$.get(Option.scala:345)
        at
org.apache.spark.sql.carbondata.execution.datasources.SparkCarbonFileFormat.<init>(SparkCarbonFileFormat.scala:120)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at java.lang.Class.newInstance(Class.java:442)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
        ... 17 more

  Thanks.



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[ravipesala]

Hi,

I will check and fix it.

Regards,
Ravindra

On Fri, 16 Nov 2018 at 09:24, xm_zzc <[hidden email]> wrote:

> Hi:
>   Please help. I used CarbonSession to read parquet and it throws
> 'NoSuchElementException: None.get' error, reading carbondata files is ok.
>   *Env*: local mode, Spark 2.3 + CarbonData(master branch)
>   *Code*:
>     import org.apache.spark.sql.CarbonSession._
>     val spark = SparkSession
>       .builder()
>       .master("local[1]")
>       .appName("Carbon1_5")
>       .config("spark.sql.warehouse.dir", warehouse)
>       .config("spark.default.parallelism", 4)
>       .config("spark.sql.shuffle.partitions", 4)
>       .getOrCreateCarbonSession(storeLocation, Constants.METASTORE_DB)
>     spark.conf.set("spark.sql.parquet.binaryAsString", true)
>     val parquets = spark.read.parquet("/data1/parquets/")
>     println(parquets.count())
>
>   *Error*:
>     Exception in thread "main" java.util.ServiceConfigurationError:
> org.apache.spark.sql.sources.DataSourceRegister: Provider
> org.apache.spark.sql.carbondata.execution.datasources.SparkCarbonFileFormat
> could not be instantiated
>         at java.util.ServiceLoader.fail(ServiceLoader.java:232)
>         at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
>         at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
>         at
> java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>         at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>         at
> scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:742)
>         at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
>         at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>         at
>
> scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:258)
>         at
> scala.collection.TraversableLike$class.filter(TraversableLike.scala:270)
>         at
> scala.collection.AbstractTraversable.filter(Traversable.scala:104)
>         at
>
> org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:618)
>         at
> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:190)
>         at
> org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:622)
>         at
> org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:606)
>         at
> cn.xm.zzc.carbonmaster.Carbon1_5$.testReadSpeed(Carbon1_5.scala:434)
>         at cn.xm.zzc.carbonmaster.Carbon1_5$.main(Carbon1_5.scala:105)
>         at cn.xm.zzc.carbonmaster.Carbon1_5.main(Carbon1_5.scala)
> Caused by: java.util.NoSuchElementException: None.get
>         at scala.None$.get(Option.scala:347)
>         at scala.None$.get(Option.scala:345)
>         at
>
> org.apache.spark.sql.carbondata.execution.datasources.SparkCarbonFileFormat.<init>(SparkCarbonFileFormat.scala:120)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>         at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>         at java.lang.Class.newInstance(Class.java:442)
>         at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
>         ... 17 more
>
>   Thanks.
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>

--
Thanks & Regards,
Ravi

[xm_zzc]

PR#2863 has fixed this issue, thanks, Ravindra.



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/