Operation not allowed: STORED BY (from Spark Dataframe save)

Posted by yannv on Aug 14, 2018; 3:14pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Operation-not-allowed-STORED-BY-from-Spark-Dataframe-save-tp59584.html

Hello,

I am trying to create a carbon data table from a Spark Data Frame, however I
am getting an error with the (automatic create table statement)

I run this code on spark-shell (passing the carbon data assembly jar file
for 1.4.0 as well as master branch), on Azure HDInsight cluster with spark
2.2.1.

Code used :


org.apache.spark.sql.catalyst.parser.ParseException:
Operation not allowed: STORED BY(line 5, pos 1)

== SQL ==

 CREATE TABLE IF NOT EXISTS default.carbon_df_table_test1
 (c2 STRING, number INT)
 PARTITIONED BY (c1 string)
 STORED BY 'carbondata'
-^^^

  TBLPROPERTIES ('STREAMING' = 'false')



  at
org.apache.spark.sql.catalyst.parser.ParserUtils$.operationNotAllowed(ParserUtils.scala:39)
  at
org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateFileFormat$1.apply(SparkSqlParser.scala:1194)
  at
org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateFileFormat$1.apply(SparkSqlParser.scala:1186)
  at
org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:99)
  at
org.apache.spark.sql.execution.SparkSqlAstBuilder.visitCreateFileFormat(SparkSqlParser.scala:1185)
  at
org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateHiveTable$1$$anonfun$31.apply(SparkSqlParser.scala:1090)
  at
org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateHiveTable$1$$anonfun$31.apply(SparkSqlParser.scala:1090)
  at scala.Option.map(Option.scala:146)
  at
org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateHiveTable$1.apply(SparkSqlParser.scala:1090)



I tried various constructors for the carbon object without success.

Note : I can create a Carbondata table and insert data from CSV file
successfully (but I need to write carbon data from SparkDF), but it looks
like when the save method is executed, it tries to create the (new) table
and I get this error on "Stored by"...



Regards,
Yann



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/