GitHub user SteNicholas opened a pull request:
https://github.com/apache/carbondata/pull/3012 [CARBONDATA-3127]Fix the HiveExample & TestCarbonSerde exception [CARBONDATA-3127]Hive module test case has been commented off,can' t run. This pull request fix TestCarbonSerde test case exception with maven compile.Method called deserializeAndSerializeLazySimple of TestCarbonSerde class calls the interface called serializeStartKey which is not exsit in CarbonHiveSerDe class.Just modify replace serializeStartKey with serialize and remove comments of TestCarbonSerde. Previous hive-integration HiveExample exists obvious bug that there is no data when select HIVE_CARBON_EXAMPLE table.I have already fixed the bug above in this pull request.Location was configured wrong after creating hive table.And block lack of detail info and footer offset when do querying hive table operation.Therefore I set detail info of block with footer offset through input split. - [x] Any interfaces changed? No - [x] Any backward compatibility impacted? No - [x] Document update required? No - [x] Testing done Modify TestCarbonSerde test - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. No You can merge this pull request into a Git repository by running: $ git pull https://github.com/SteNicholas/carbondata CARBONDATA-3127 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/3012.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3012 ---- commit f403455748aa98a224559dac7e8953725d98b41c Author: Nicholas Jiang <programgeek@...> Date: 2018-12-20T20:14:09Z Hive Integration ---- --- |
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1889/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2098/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10143/ --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243470291 --- Diff: examples/spark2/pom.xml --- @@ -78,7 +83,12 @@ <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> - <version>${httpclient.version}</version> + <version>4.3.4</version> --- End diff -- Please use the unify version --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243470296 --- Diff: examples/spark2/pom.xml --- @@ -78,7 +83,12 @@ <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> - <version>${httpclient.version}</version> + <version>4.3.4</version> + </dependency> + <dependency> + <groupId>org.apache.httpcomponents</groupId> + <artifactId>httpcore</artifactId> + <version>4.3-alpha1</version> --- End diff -- Please use the unify version --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243470427 --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/HiveExample.scala --- @@ -30,44 +33,38 @@ object HiveExample { private val driverName: String = "org.apache.hive.jdbc.HiveDriver" def main(args: Array[String]) { - val rootPath = new File(this.getClass.getResource("/").getPath - + "../../../..").getCanonicalPath - val store = s"$rootPath/integration/hive/target/store" - val warehouse = s"$rootPath/integration/hive/target/warehouse" - val metaStore_Db = s"$rootPath/integration/hive/target/carbon_metaStore_db" - val logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName) - var resultId = "" - var resultName = "" - var resultSalary = "" - + val carbonSession = ExampleUtils.createCarbonSession("HiveExample") + exampleBody(carbonSession, CarbonProperties.getStorePath + + CarbonCommonConstants.FILE_SEPARATOR + + CarbonCommonConstants.DATABASE_DEFAULT_NAME) + carbonSession.close() - import org.apache.spark.sql.CarbonSession._ + System.exit(0) + } - val carbonSession = SparkSession - .builder() - .master("local") - .appName("HiveExample") - .config("carbonSession.sql.warehouse.dir", warehouse).enableHiveSupport() - .getOrCreateCarbonSession( - store, metaStore_Db) + def exampleBody(sparkSession: SparkSession, store: String): Unit = { --- End diff -- carbonSession is better --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243470594 --- Diff: examples/spark2/src/test/scala/org/apache/carbondata/examplesCI/RunExamples.scala --- @@ -19,12 +19,12 @@ package org.apache.carbondata.examplesCI import org.apache.spark.sql.test.util.QueryTest import org.scalatest.BeforeAndAfterAll - import org.apache.carbondata.examples._ import org.apache.carbondata.core.constants.CarbonCommonConstants import org.apache.carbondata.core.util.CarbonProperties import org.apache.carbondata.examples.sdk.CarbonReaderExample import org.apache.carbondata.examples.sql.JavaCarbonSessionExample +import org.apache.spark.sql.test.TestQueryExecutor --- End diff -- please move o another group --- |
In reply to this post by qiuchenjian-2
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243478043 --- Diff: examples/spark2/pom.xml --- @@ -78,7 +83,12 @@ <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> - <version>${httpclient.version}</version> + <version>4.3.4</version> --- End diff -- @xubo245 I have already use the unify version of http client library. --- |
In reply to this post by qiuchenjian-2
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243478363 --- Diff: examples/spark2/pom.xml --- @@ -78,7 +83,12 @@ <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> - <version>${httpclient.version}</version> + <version>4.3.4</version> + </dependency> + <dependency> + <groupId>org.apache.httpcomponents</groupId> + <artifactId>httpcore</artifactId> + <version>4.3-alpha1</version> --- End diff -- @xubo245 I have already use the unify version of http core library. --- |
In reply to this post by qiuchenjian-2
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243478424 --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/HiveExample.scala --- @@ -30,44 +33,38 @@ object HiveExample { private val driverName: String = "org.apache.hive.jdbc.HiveDriver" def main(args: Array[String]) { - val rootPath = new File(this.getClass.getResource("/").getPath - + "../../../..").getCanonicalPath - val store = s"$rootPath/integration/hive/target/store" - val warehouse = s"$rootPath/integration/hive/target/warehouse" - val metaStore_Db = s"$rootPath/integration/hive/target/carbon_metaStore_db" - val logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName) - var resultId = "" - var resultName = "" - var resultSalary = "" - + val carbonSession = ExampleUtils.createCarbonSession("HiveExample") + exampleBody(carbonSession, CarbonProperties.getStorePath + + CarbonCommonConstants.FILE_SEPARATOR + + CarbonCommonConstants.DATABASE_DEFAULT_NAME) + carbonSession.close() - import org.apache.spark.sql.CarbonSession._ + System.exit(0) + } - val carbonSession = SparkSession - .builder() - .master("local") - .appName("HiveExample") - .config("carbonSession.sql.warehouse.dir", warehouse).enableHiveSupport() - .getOrCreateCarbonSession( - store, metaStore_Db) + def exampleBody(sparkSession: SparkSession, store: String): Unit = { --- End diff -- @xubo245 I update parameter name in exampleBody method according to your suggestion. --- |
In reply to this post by qiuchenjian-2
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243478610 --- Diff: examples/spark2/src/test/scala/org/apache/carbondata/examplesCI/RunExamples.scala --- @@ -19,12 +19,12 @@ package org.apache.carbondata.examplesCI import org.apache.spark.sql.test.util.QueryTest import org.scalatest.BeforeAndAfterAll - import org.apache.carbondata.examples._ import org.apache.carbondata.core.constants.CarbonCommonConstants import org.apache.carbondata.core.util.CarbonProperties import org.apache.carbondata.examples.sdk.CarbonReaderExample import org.apache.carbondata.examples.sql.JavaCarbonSessionExample +import org.apache.spark.sql.test.TestQueryExecutor --- End diff -- @xubo245 Yeah, I move to another group.TestCarbonSerDe is also updated as same as the way of RunExamples.scala. --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1892/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3012 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10147/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3012 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2101/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1893/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10148/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2102/ --- |
In reply to this post by qiuchenjian-2
|
In reply to this post by qiuchenjian-2
|
Free forum by Nabble | Edit this page |