method not found issue when creating table

[lionel061201]

Hi dev,
When I was trying to create table in carbon, below error occurred, anyone
knows how to fix?

1 driver, 6 datanode

*$SPARK_HOME* = /opt/cloudera/parcels/SPARK2/lib/spark2/

*configuration in spark-defaults.conf:*

#Carbondata Settings

spark.yarn.dist.files=/opt/cloudera/parcels/SPARK2/lib/spark2/conf/carbon.properties

spark.yarn.dist.archives=/opt/cloudera/parcels/SPARK2/lib/spark2/carbonlib/carbondata.tar.gz

spark.executor.extraClassPath=/opt/cloudera/parcels/SPARK2/lib/spark2/carbonlib/*

spark.driver.extraClassPath=/opt/cloudera/parcels/SPARK2/lib/spark2/carbonlib/*

*carbon jar (distributed in all driver and datanodes)*

/opt/cloudera/parcels/SPARK2/lib/spark2/carbonlib/carbondata_2.11-1.2.0-SNAPSHOT-shade-hadoop2.2.0.jar

spark2-shell --master yarn --deploy-mode client --num-executors 6
--driver-memory 20G --executor-memory 50G --executor-cores 4


scala> import org.apache.spark.sql.CarbonSession._

scala> import org.apache.spark.sql.SparkSession

scala> import org.apache.carbondata.core.util.CarbonProperties

scala> import org.apache.carbondata.core.constants.CarbonCommonConstants

scala> val cc =
SparkSession.builder().appName("CL_TEST").config(sc.getConf).getOrCreateCarbonSession("hdfs://nameservice2/carbon2/carbonstore")

scala> CarbonProperties.*getInstance*().addProperty(CarbonCommonConstants.
*CARBON_TIMESTAMP_FORMAT*, "yyyy-MM-dd")

scala> cc.sql("use default")

scala> *cc.sql("create table test001(vin string) stored by
'carbondata'").show()*

17/08/22 14:34:35 AUDIT command.CreateTable:
[****.*****.com][carbon2][Thread-1]Creating Table with Database name
[default] and Table name [test001]

*java.lang.NoSuchMethodError:
org.apache.spark.sql.catalyst.catalog.CatalogTable.copy(Lorg/apache/spark/sql/catalyst/TableIdentifier;Lorg/apache/spark/sql/catalyst/catalog/CatalogTableType;Lorg/apache/spark/sql/catalyst/catalog/CatalogStorageFormat;Lorg/apache/spark/sql/types/StructType;Lscala/Option;Lscala/collection/Seq;Lscala/Option;Ljava/lang/String;JJLscala/collection/immutable/Map;Lscala/Option;Lscala/Option;Lscala/Option;Lscala/Option;Lscala/collection/Seq;Z)Lorg/apache/spark/sql/catalyst/catalog/CatalogTable;*

  at
org.apache.spark.sql.CarbonSource$.updateCatalogTableWithCarbonSchema(CarbonSource.scala:277)

  at
org.apache.spark.sql.execution.command.DDLStrategy.apply(DDLStrategy.scala:135)

  at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:62)

  at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:62)

  at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)

  at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)

  at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)

  at
org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:92)

  at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:77)

  at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:74)

  at
scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)

  at
scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)

  at scala.collection.Iterator$class.foreach(Iterator.scala:893)

  at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)

  at
scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)

  at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1336)

  at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:74)

  at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2.apply(QueryPlanner.scala:66)

  at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)

  at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)

  at
org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:92)

  at
org.apache.spark.sql.execution.QueryExecution.sparkPlan$lzycompute(QueryExecution.scala:79)

  at
org.apache.spark.sql.execution.QueryExecution.sparkPlan(QueryExecution.scala:75)

  at
org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:84)

  at
org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:84)

  at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)

  at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)

  at org.apache.spark.sql.Dataset.<init>(Dataset.scala:185)

  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)

  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)

  at
org.apache.spark.sql.execution.command.CreateTable.processSchema(carbonTableSchema.scala:496)

  at
org.apache.spark.sql.execution.command.CreateTable.run(carbonTableSchema.scala:452)

  at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)

  at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)

  at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)

  at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)

  at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)

  at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)

  at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

  at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)

  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)

  at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)

  at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)

  at org.apache.spark.sql.Dataset.<init>(Dataset.scala:185)

  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)

  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)

  ... 50 elided

[manishgupta88]

Hi Lionel,

Carbon table creation flow is executed on the driver side, Executors do not participate in creation of carbon table. From the logs it seems that spark-catalyst jar is missing which is generally placed under $SPARK_HOME/jars OR $SPARK_HOME/lib directory. Please check if spark jars directory is there in the driver classpath. You can follow the below steps:

1. On the driver node execute "jps" comamnd and find out the SparkSubmit process id.
2. Execute "jinfo <process id> and redirect it to a file.
3. Search for spark-catalyst jar in the file. If not found that means its not in the classpath and you can then add the jar in classpath and run your queries again.

Regards
Manish Gupta

[lionel061201]

Hi Manish,
Thank you for the response!
I have addressed the issue is caused by the spark catalyst jar of cloudera
version is different from the open source version. The case class
CatalogTable in cloudera version has some more columns.

But how can I change the instance of CatalogTable(tableDesc in the
following codes) to the cloudera version? it comes from a SparkPlan...
Though I have changed all the jar dependecies to cloudera version, it still
return the No Such Method error.


/**
 * Carbon strategies for ddl commands
 */
class DDLStrategy(sparkSession: SparkSession) extends SparkStrategy {

  def apply(plan: LogicalPlan): Seq[SparkPlan] = {
    plan match {
      case ...
      case
org.apache.spark.sql.execution.datasources.CreateTable(tableDesc, mode,
None)
        if tableDesc.provider.get != DDLUtils.HIVE_PROVIDER
           &&
tableDesc.provider.get.equals("org.apache.spark.sql.CarbonSource") =>
        val updatedCatalog =
          CarbonSource.updateCatalogTableWithCarbonSchema(tableDesc,
sparkSession)
        val cmd =
          CreateDataSourceTableCommand(updatedCatalog, ignoreIfExists =
mode == SaveMode.Ignore)
        ExecutedCommandExec(cmd) :: Nil
      case _ => Nil
    }
  }
}

the cloudera version case class is

case class CatalogTable(
    identifier: TableIdentifier,
    tableType: CatalogTableType,
    storage: CatalogStorageFormat,
    schema: StructType,
    provider: Option[String] = None,
    partitionColumnNames: Seq[String] = Seq.empty,
    bucketSpec: Option[BucketSpec] = None,
    owner: String = "",
    createTime: Long = System.currentTimeMillis,
    lastAccessTime: Long = -1,
    properties: Map[String, String] = Map.empty,
    stats: Option[Statistics] = None,
    viewOriginalText: Option[String] = None,
    viewText: Option[String] = None,
    comment: Option[String] = None,
    unsupportedFeatures: Seq[String] = Seq.empty,
    tracksPartitionsInCatalog: Boolean = false,
    schemaPreservesCase: Boolean = true)

while in the error log, copy method is

java.lang.NoSuchMethodError:
org.apache.spark.sql.catalyst.catalog.CatalogTable.copy(
Lorg/apache/spark/sql/catalyst/TableIdentifier;
Lorg/apache/spark/sql/catalyst/catalog/CatalogTableType;
Lorg/apache/spark/sql/catalyst/catalog/CatalogStorageFormat;
Lorg/apache/spark/sql/types/StructType;
Lscala/Option;
Lscala/collection/Seq;
Lscala/Option;
Ljava/lang/String;
  <<<--------- Miss two long type parameters here
JJLscala/collection/immutable/Map;
Lscala/Option;
Lscala/Option;
Lscala/Option;
Lscala/Option;
Lscala/collection/Seq;Z)

Thanks,
CaoLu

On Wed, Aug 23, 2017 at 5:10 PM, manishgupta88 <[hidden email]>
wrote:

> Hi Lionel,
>
> Carbon table creation flow is executed on the driver side, Executors do not
> participate in creation of carbon table. From the logs it seems that
> spark-catalyst jar is missing which is generally placed under
> $SPARK_HOME/jars OR $SPARK_HOME/lib directory. Please check if spark jars
> directory is there in the driver classpath. You can follow the below steps:
>
> 1. On the driver node execute "jps" comamnd and find out the SparkSubmit
> process id.
> 2. Execute "jinfo <process id> and redirect it to a file.
> 3. Search for spark-catalyst jar in the file. If not found that means its
> not in the classpath and you can then add the jar in classpath and run your
> queries again.
>
> Regards
> Manish Gupta
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/method-not-
> found-issue-when-creating-table-tp20640p20702.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>