Hi all,
I've tried the following two ways to compile carbondata, but they finally all failed, if anyone could help me: 1. with the latest verson carbondata at github and spark 2.0.0, use the command mvn -DskipTests -Pspark-2.0 -Dspark.version=2.0.0 clean package and it compiled successly. but when I run the script ./bin/carbon-spark-shell , it turns out: ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory ./bin/carbon-spark-shell: line 78: /bin/spark-submit: No such file or directory and when I run the script ./bin/carbon-spark-sql, it turns out: $SPARK_HOME is not set ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory ./bin/carbon-spark-sql: line 79: cmd: command not found ./bin/carbon-spark-sql: line 89: /bin/spark-submit: No such file or directory ./bin/carbon-spark-sql: line 89: exec: /bin/spark-submit: cannot execute: No such file or directory 2.with the carbondata-0.2.0 and spark 1.5.0, use the command mvn -DskipTests -Pspark-1.5 -X -Dspark.version=1.5.0 clean package and it compiled failed. the error message is : [ERROR] /home/hadoop/carbondata-0.2.0/integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetadataUtil.scala:46: error: not found: value SqlParser [INFO] val tableIdent = SqlParser.parseTableIdentifier(tableWithDb) [INFO] ^ [WARNING] one warning found [ERROR] one error found [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache CarbonData :: Parent ........................ SUCCESS [ 0.971 s] [INFO] Apache CarbonData :: Common ........................ SUCCESS [ 1.702 s] [INFO] Apache CarbonData :: Core .......................... SUCCESS [ 4.428 s] [INFO] Apache CarbonData :: Processing .................... SUCCESS [ 1.587 s] [INFO] Apache CarbonData :: Hadoop ........................ SUCCESS [ 1.104 s] [INFO] Apache CarbonData :: Spark ......................... FAILURE [ 11.456 s] [INFO] Apache CarbonData :: Assembly ...................... SKIPPED [INFO] Apache CarbonData :: Examples ...................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 21.432 s [INFO] Finished at: 2016-12-15T19:41:29+08:00 [INFO] Final Memory: 69M/995M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) Caused by: org.apache.maven.plugin.MojoExecutionException: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:350) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) ... 20 more Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:346) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:149) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:136) at org_scala_tools_maven_executions.JavaMainCallerByFork.run(JavaMainCallerByFork.java:80) at org_scala_tools_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:124) at org_scala_tools_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:80) at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:342) ... 22 more [ERROR] [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :carbondata-spark |
Carbon don't support spark 2.x
------------------ Original ------------------ From: "李寅威";<[hidden email]>; Date: Thu, Dec 15, 2016 07:45 PM To: "dev"<[hidden email]>; Subject: Some questions about compiling carbondata Hi all, I've tried the following two ways to compile carbondata, but they finally all failed, if anyone could help me: 1. with the latest verson carbondata at github and spark 2.0.0, use the command mvn -DskipTests -Pspark-2.0 -Dspark.version=2.0.0 clean package and it compiled successly. but when I run the script ./bin/carbon-spark-shell , it turns out: ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory ./bin/carbon-spark-shell: line 78: /bin/spark-submit: No such file or directory and when I run the script ./bin/carbon-spark-sql, it turns out: $SPARK_HOME is not set ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory ./bin/carbon-spark-sql: line 79: cmd: command not found ./bin/carbon-spark-sql: line 89: /bin/spark-submit: No such file or directory ./bin/carbon-spark-sql: line 89: exec: /bin/spark-submit: cannot execute: No such file or directory 2.with the carbondata-0.2.0 and spark 1.5.0, use the command mvn -DskipTests -Pspark-1.5 -X -Dspark.version=1.5.0 clean package and it compiled failed. the error message is : [ERROR] /home/hadoop/carbondata-0.2.0/integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetadataUtil.scala:46: error: not found: value SqlParser [INFO] val tableIdent = SqlParser.parseTableIdentifier(tableWithDb) [INFO] ^ [WARNING] one warning found [ERROR] one error found [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache CarbonData :: Parent ........................ SUCCESS [ 0.971 s] [INFO] Apache CarbonData :: Common ........................ SUCCESS [ 1.702 s] [INFO] Apache CarbonData :: Core .......................... SUCCESS [ 4.428 s] [INFO] Apache CarbonData :: Processing .................... SUCCESS [ 1.587 s] [INFO] Apache CarbonData :: Hadoop ........................ SUCCESS [ 1.104 s] [INFO] Apache CarbonData :: Spark ......................... FAILURE [ 11.456 s] [INFO] Apache CarbonData :: Assembly ...................... SKIPPED [INFO] Apache CarbonData :: Examples ...................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 21.432 s [INFO] Finished at: 2016-12-15T19:41:29+08:00 [INFO] Final Memory: 69M/995M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) Caused by: org.apache.maven.plugin.MojoExecutionException: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:350) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) ... 20 more Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:346) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:149) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:136) at org_scala_tools_maven_executions.JavaMainCallerByFork.run(JavaMainCallerByFork.java:80) at org_scala_tools_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:124) at org_scala_tools_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:80) at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:342) ... 22 more [ERROR] [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :carbondata-spark |
In reply to this post by 李寅威
Hi,
You do not need to specify spark.version variable, you can try these: mvn clean package -DskipTests -Pspark-2.0 (to build carbon with spark-2.0.2) mvn clean package -DskipTests (to build carbon with spark-1.5.2, which is default profile) Regards, Jacky |
In reply to this post by cenyuhai
请问这个问题现在解决了么?
|
In reply to this post by 李寅威
Hi
For Question one, I have raised discussion about carbon-spark-shell for spark2.x in following link. http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-Whether-Carbondata-should-keep-carbon-spark-shell-script-td14077.html Actually there is PR to fix carbon-spark-shell for spark2.x, but I think this script is useless. You can give your opinion, we can vote for this. Regards. Chenerlu |
Carbon does not support Spark 2.0, please use Spark 2.1
Use the following command to compile mvn clean -Pspark-2.1 package Regards, Ravindra. On 15 June 2017 at 14:44, Erlu Chen <[hidden email]> wrote: > Hi > > For Question one, I have raised discussion about carbon-spark-shell for > spark2.x in following link. > > http://apache-carbondata-dev-mailing-list-archive.1130556. > n5.nabble.com/DISCUSSION-Whether-Carbondata-should- > keep-carbon-spark-shell-script-td14077.html > > Actually there is PR to fix carbon-spark-shell for spark2.x, but I think > this script is useless. > > You can give your opinion, we can vote for this. > > Regards. > Chenerlu > > > > -- > View this message in context: http://apache-carbondata-dev- > mailing-list-archive.1130556.n5.nabble.com/Some-questions- > about-compiling-carbondata-tp4498p15139.html > Sent from the Apache CarbonData Dev Mailing List archive mailing list > archive at Nabble.com. > -- Thanks & Regards, Ravi |
Free forum by Nabble | Edit this page |