Re: Operation not allowed: STORED BY (from Spark Dataframe save)

Posted by sraghunandan on Aug 14, 2018; 3:35pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Operation-not-allowed-STORED-BY-from-Spark-Dataframe-save-tp59584p59585.html

Hi yannav,
Can you send the df api and the code you have used.
Did you refer the example in TestLoadDataFrame.scala?

Are you trying from spark session or carbonsession?

Regards
Raghu

On Tue, 14 Aug 2018, 8:44 pm yannv, <[hidden email]> wrote:

> Hello,
>
> I am trying to create a carbon data table from a Spark Data Frame, however
> I
> am getting an error with the (automatic create table statement)
>
> I run this code on spark-shell (passing the carbon data assembly jar file
> for 1.4.0 as well as master branch), on Azure HDInsight cluster with spark
> 2.2.1.
>
> Code used :
>
>
> org.apache.spark.sql.catalyst.parser.ParseException:
> Operation not allowed: STORED BY(line 5, pos 1)
>
> == SQL ==
>
>  CREATE TABLE IF NOT EXISTS default.carbon_df_table_test1
>  (c2 STRING, number INT)
>  PARTITIONED BY (c1 string)
>  STORED BY 'carbondata'
> -^^^
>
>   TBLPROPERTIES ('STREAMING' = 'false')
>
>
>
>   at
>
> org.apache.spark.sql.catalyst.parser.ParserUtils$.operationNotAllowed(ParserUtils.scala:39)
>   at
>
> org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateFileFormat$1.apply(SparkSqlParser.scala:1194)
>   at
>
> org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateFileFormat$1.apply(SparkSqlParser.scala:1186)
>   at
>
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:99)
>   at
>
> org.apache.spark.sql.execution.SparkSqlAstBuilder.visitCreateFileFormat(SparkSqlParser.scala:1185)
>   at
>
> org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateHiveTable$1$$anonfun$31.apply(SparkSqlParser.scala:1090)
>   at
>
> org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateHiveTable$1$$anonfun$31.apply(SparkSqlParser.scala:1090)
>   at scala.Option.map(Option.scala:146)
>   at
>
> org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateHiveTable$1.apply(SparkSqlParser.scala:1090)
>
>
>
> I tried various constructors for the carbon object without success.
>
> Note : I can create a Carbondata table and insert data from CSV file
> successfully (but I need to write carbon data from SparkDF), but it looks
> like when the save method is executed, it tries to create the (new) table
> and I get this error on "Stored by"...
>
>
>
> Regards,
> Yann
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>