after load data using SaveMode.Overwrite, query through beeline return all null field

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

after load data using SaveMode.Overwrite, query through beeline return all null field

喜之郎
hi dev.
carbon version :1.3.1
spark version:2.2.1
1) First I create a carbon table through beeline.
2) Then I use spark-submit and dataframe load data to carbon. Query is OK。
3) Then I use spark-submit and dataframe load data to carbon again, but query through thriftserver return all NULL field.
4) I restart thriftserver, and query through thriftserver return correct result.


This problem happens everytime. I think this problem may is about CarbonThriftServer.
is there anybody encountered this problem? can anybody give some advice,thanks.


you can use this code to reproduce problem.
===================



val carbonTableName = args(1)

val warehouse = new File("./warehouse").getCanonicalPath
val metastore = new File("./metastore").getCanonicalPath
val spark = SparkSession
  .builder()
  .appName("StreamExample")
  .config("spark.sql.warehouse.dir", warehouse)
  .getOrCreateCarbonSession(warehouse, metastore)
val df2 = spark.read.parquet(path)
df2.write
  .format("carbondata")
  .option("tableName", carbonTableName)
  .option("compress", "true")
  .option("tempCSV", "false")
  .mode(SaveMode.Overwrite)
  .save()=============
Reply | Threaded
Open this post in threaded view
|

Re: after load data using SaveMode.Overwrite, query through beeline return all null field

Liang Chen
Administrator
Hi

Thank you reported this issue.
Let us check it and response to you asap.

Regards
Liang


喜之郎 wrote

> hi dev.
> carbon version :1.3.1
> spark version:2.2.1
> 1) First I create a carbon table through beeline.
> 2) Then I use spark-submit and dataframe load data to carbon. Query is OK。
> 3) Then I use spark-submit and dataframe load data to carbon again, but
> query through thriftserver return all NULL field.
> 4) I restart thriftserver, and query through thriftserver return correct
> result.
>
>
> This problem happens everytime. I think this problem may is about
> CarbonThriftServer.
> is there anybody encountered this problem? can anybody give some
> advice,thanks.
>
>
> you can use this code to reproduce problem.
> ===================
>
>
>
> val carbonTableName = args(1)
>
> val warehouse = new File("./warehouse").getCanonicalPath
> val metastore = new File("./metastore").getCanonicalPath
> val spark = SparkSession
>   .builder()
>   .appName("StreamExample")
>   .config("spark.sql.warehouse.dir", warehouse)
>   .getOrCreateCarbonSession(warehouse, metastore)
> val df2 = spark.read.parquet(path)
> df2.write
>   .format("carbondata")
>   .option("tableName", carbonTableName)
>   .option("compress", "true")
>   .option("tempCSV", "false")
>   .mode(SaveMode.Overwrite)
>   .save()=============





--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: after load data using SaveMode.Overwrite, query through beeline return all null field

akashrn5
In reply to this post by 喜之郎
Hi,

I have checked with the current version and the issue is not reproducing,
when I checked the code, there are code changes happened for the savemode
from 1.3 to 1.4 version.

You can check the PR #2186 for the changes done for that part and you can
check your issue again with that PR .


Regards,
Akash



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/