Apache CarbonData Dev Mailing List archive
Login  Register

Some questions about compiling carbondata

classic Classic list List threaded Threaded
♦
♦
 6 messages Options Options
 
Embed post
Permalink
李寅威
Reply | Threaded
Open this post in threaded view
♦
♦
| More
Print post
Permalink
Dec 15, 2016; 11:45am

Some questions about compiling carbondata

李寅威
Hi all,

  I've tried the following two ways to compile carbondata, but they finally all failed, if anyone could help me:


  1. with the latest verson carbondata at github and spark 2.0.0, use the command


      mvn -DskipTests -Pspark-2.0 -Dspark.version=2.0.0 clean package


     and it compiled successly.


      but when I run the script   ./bin/carbon-spark-shell  ,  it turns out:
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ./bin/carbon-spark-shell: line 78: /bin/spark-submit: No such file or directory


      and when I run the script  ./bin/carbon-spark-sql, it turns out:
      $SPARK_HOME is not set
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ./bin/carbon-spark-sql: line 79: cmd: command not found
      ./bin/carbon-spark-sql: line 89: /bin/spark-submit: No such file or directory
      ./bin/carbon-spark-sql: line 89: exec: /bin/spark-submit: cannot execute: No such file or directory


    2.with the carbondata-0.2.0 and spark 1.5.0, use the command


      mvn -DskipTests -Pspark-1.5 -X -Dspark.version=1.5.0 clean package


     and it compiled failed. the error message is :


        [ERROR] /home/hadoop/carbondata-0.2.0/integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetadataUtil.scala:46: error: not found: value SqlParser
        [INFO]     val tableIdent = SqlParser.parseTableIdentifier(tableWithDb)
        [INFO]                      ^
        [WARNING] one warning found
        [ERROR] one error found
        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO]
        [INFO] Apache CarbonData :: Parent ........................ SUCCESS [  0.971 s]
        [INFO] Apache CarbonData :: Common ........................ SUCCESS [  1.702 s]
        [INFO] Apache CarbonData :: Core .......................... SUCCESS [  4.428 s]
        [INFO] Apache CarbonData :: Processing .................... SUCCESS [  1.587 s]
        [INFO] Apache CarbonData :: Hadoop ........................ SUCCESS [  1.104 s]
        [INFO] Apache CarbonData :: Spark ......................... FAILURE [ 11.456 s]
        [INFO] Apache CarbonData :: Assembly ...................... SKIPPED
        [INFO] Apache CarbonData :: Examples ...................... SKIPPED
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD FAILURE
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 21.432 s
        [INFO] Finished at: 2016-12-15T19:41:29+08:00
        [INFO] Final Memory: 69M/995M
        [INFO] ------------------------------------------------------------------------
        [ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) -> [Help 1]
        org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
                at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
                at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
                at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
                at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
                at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
                at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
                at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
                at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
                at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
                at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:497)
                at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
                at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
                at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
                at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
        Caused by: org.apache.maven.plugin.MojoExecutionException: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
                at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:350)
                at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
                ... 20 more
        Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
                at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:346)
                at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:149)
                at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:136)
                at org_scala_tools_maven_executions.JavaMainCallerByFork.run(JavaMainCallerByFork.java:80)
                at org_scala_tools_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:124)
                at org_scala_tools_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:80)
                at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:342)
                ... 22 more
        [ERROR]
        [ERROR]
        [ERROR] For more information about the errors and possible solutions, please read the following articles:
        [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
        [ERROR]
        [ERROR] After correcting the problems, you can resume the build with the command
        [ERROR]   mvn <goals> -rf :carbondata-spark
cenyuhai
Reply | Threaded
Open this post in threaded view
♦
♦
| More
Print post
Permalink
Dec 15, 2016; 1:12pm

Re: Some questions about compiling carbondata

cenyuhai
Carbon don't support spark 2.x




------------------ Original ------------------
From:  "李寅威";<[hidden email]>;
Date:  Thu, Dec 15, 2016 07:45 PM
To:  "dev"<[hidden email]>;

Subject:  Some questions about compiling carbondata



Hi all,

  I've tried the following two ways to compile carbondata, but they finally all failed, if anyone could help me:


  1. with the latest verson carbondata at github and spark 2.0.0, use the command


      mvn -DskipTests -Pspark-2.0 -Dspark.version=2.0.0 clean package


     and it compiled successly.


      but when I run the script   ./bin/carbon-spark-shell  ,  it turns out:
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ./bin/carbon-spark-shell: line 78: /bin/spark-submit: No such file or directory


      and when I run the script  ./bin/carbon-spark-sql, it turns out:
      $SPARK_HOME is not set
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ./bin/carbon-spark-sql: line 79: cmd: command not found
      ./bin/carbon-spark-sql: line 89: /bin/spark-submit: No such file or directory
      ./bin/carbon-spark-sql: line 89: exec: /bin/spark-submit: cannot execute: No such file or directory


    2.with the carbondata-0.2.0 and spark 1.5.0, use the command


      mvn -DskipTests -Pspark-1.5 -X -Dspark.version=1.5.0 clean package


     and it compiled failed. the error message is :


        [ERROR] /home/hadoop/carbondata-0.2.0/integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetadataUtil.scala:46: error: not found: value SqlParser
        [INFO]     val tableIdent = SqlParser.parseTableIdentifier(tableWithDb)
        [INFO]                      ^
        [WARNING] one warning found
        [ERROR] one error found
        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO]
        [INFO] Apache CarbonData :: Parent ........................ SUCCESS [  0.971 s]
        [INFO] Apache CarbonData :: Common ........................ SUCCESS [  1.702 s]
        [INFO] Apache CarbonData :: Core .......................... SUCCESS [  4.428 s]
        [INFO] Apache CarbonData :: Processing .................... SUCCESS [  1.587 s]
        [INFO] Apache CarbonData :: Hadoop ........................ SUCCESS [  1.104 s]
        [INFO] Apache CarbonData :: Spark ......................... FAILURE [ 11.456 s]
        [INFO] Apache CarbonData :: Assembly ...................... SKIPPED
        [INFO] Apache CarbonData :: Examples ...................... SKIPPED
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD FAILURE
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 21.432 s
        [INFO] Finished at: 2016-12-15T19:41:29+08:00
        [INFO] Final Memory: 69M/995M
        [INFO] ------------------------------------------------------------------------
        [ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) -> [Help 1]
        org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
                at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
                at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
                at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
                at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
                at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
                at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
                at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
                at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
                at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
                at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:497)
                at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
                at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
                at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
                at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
        Caused by: org.apache.maven.plugin.MojoExecutionException: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
                at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:350)
                at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
                ... 20 more
        Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
                at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:346)
                at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:149)
                at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:136)
                at org_scala_tools_maven_executions.JavaMainCallerByFork.run(JavaMainCallerByFork.java:80)
                at org_scala_tools_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:124)
                at org_scala_tools_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:80)
                at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:342)
                ... 22 more
        [ERROR]
        [ERROR]
        [ERROR] For more information about the errors and possible solutions, please read the following articles:
        [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
        [ERROR]
        [ERROR] After correcting the problems, you can resume the build with the command
        [ERROR]   mvn <goals> -rf :carbondata-spark
Jacky Li
Reply | Threaded
Open this post in threaded view
♦
♦
| More
Print post
Permalink
Dec 15, 2016; 1:36pm

Re: Some questions about compiling carbondata

Jacky Li
In reply to this post by 李寅威
Hi,

You do not need to specify spark.version variable, you can try these:
mvn clean package -DskipTests -Pspark-2.0  (to build carbon with spark-2.0.2)
mvn clean package -DskipTests  (to build carbon with spark-1.5.2, which is default profile)

Regards,
Jacky
lzzuo2
Reply | Threaded
Open this post in threaded view
♦
♦
| More
Print post
Permalink
Jun 15, 2017; 8:20am

Re: Some questions about compiling carbondata

lzzuo2
In reply to this post by cenyuhai
请问这个问题现在解决了么？
Erlu Chen
Reply | Threaded
Open this post in threaded view
♦
♦
| More
Print post
Permalink
Jun 15, 2017; 9:14am

Re: Some questions about compiling carbondata

Erlu Chen
In reply to this post by 李寅威
Hi

For Question one, I have raised discussion about carbon-spark-shell for spark2.x in following link.

http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-Whether-Carbondata-should-keep-carbon-spark-shell-script-td14077.html

Actually there is PR to fix carbon-spark-shell for spark2.x, but I think this script is useless.

You can give your opinion, we can vote for this.

Regards.
Chenerlu
ravipesala
Reply | Threaded
Open this post in threaded view
♦
♦
| More
Print post
Permalink
Jun 15, 2017; 11:21am

Re: Some questions about compiling carbondata

ravipesala
Carbon does not support Spark 2.0, please use Spark 2.1

Use the following command to compile
   mvn clean  -Pspark-2.1  package

Regards,
Ravindra.

On 15 June 2017 at 14:44, Erlu Chen <[hidden email]> wrote:

> Hi
>
> For Question one, I have raised discussion about carbon-spark-shell for
> spark2.x in following link.
>
> http://apache-carbondata-dev-mailing-list-archive.1130556.
> n5.nabble.com/DISCUSSION-Whether-Carbondata-should-
> keep-carbon-spark-shell-script-td14077.html
>
> Actually there is PR to fix carbon-spark-shell for spark2.x, but I think
> this script is useless.
>
> You can give your opinion, we can vote for this.
>
> Regards.
> Chenerlu
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/Some-questions-
> about-compiling-carbondata-tp4498p15139.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>
... [show rest of quote]



--
Thanks & Regards,
Ravi
Free forum by Nabble Edit this page