GitHub user ManoharVanam opened a pull request:
https://github.com/apache/carbondata/pull/1683 [CARBONDATA-1911] Added Insert into query test case for parquet & hive from Carbon Table Description : Added test cases for insert into operation for parquet & hive from Carbon Table Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ManoharVanam/incubator-carbondata new Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1683.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1683 ---- ---- --- |
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1683 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2139/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1683 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2423/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1683 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/928/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1683 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2157/ --- |
In reply to this post by qiuchenjian-2
Github user ManoharVanam commented on the issue:
https://github.com/apache/carbondata/pull/1683 retest sdv please --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1683 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2447/ --- |
In reply to this post by qiuchenjian-2
Github user ManoharVanam commented on the issue:
https://github.com/apache/carbondata/pull/1683 retest sdv please --- |
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1683#discussion_r158191551 --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/insertQuery/InsertIntoNonCarbonTableTestCase.scala --- @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.carbondata.spark.testsuite.insertQuery + +import org.apache.spark.sql.Row +import org.apache.spark.sql.test.util.QueryTest +import org.scalatest.BeforeAndAfterAll + + +class InsertIntoNonCarbonTableTestCase extends QueryTest with BeforeAndAfterAll { + override def beforeAll { + sql("drop table if exists TCarbonSource") + sql( + "create table TCarbonSource (imei string,deviceInformationId int,MAC string,deviceColor " + + "string,device_backColor string,modelId string,marketName string,AMSize string,ROMSize " + + "string,CUPAudit string,CPIClocked string,series string,productionDate timestamp,bomCode " + + "string,internalModels string, deliveryTime string, channelsId string, channelsName string " + + ", deliveryAreaId string, deliveryCountry string, deliveryProvince string, deliveryCity " + + "string,deliveryDistrict string, deliveryStreet string, oxSingleNumber string, " + + "ActiveCheckTime string, ActiveAreaId string, ActiveCountry string, ActiveProvince string, " + + "Activecity string, ActiveDistrict string, ActiveStreet string, ActiveOperatorId string, " + + "Active_releaseId string, Active_EMUIVersion string, Active_operaSysVersion string, " + + "Active_BacVerNumber string, Active_BacFlashVer string, Active_webUIVersion string, " + + "Active_webUITypeCarrVer string,Active_webTypeDataVerNumber string, Active_operatorsVersion" + + " string, Active_phonePADPartitionedVersions string, Latest_YEAR int, Latest_MONTH int, " + + "Latest_DAY Decimal(30,10), Latest_HOUR string, Latest_areaId string, Latest_country " + + "string, Latest_province string, Latest_city string, Latest_district string, Latest_street " + + "string, Latest_releaseId string, Latest_EMUIVersion string, Latest_operaSysVersion string," + + " Latest_BacVerNumber string, Latest_BacFlashVer string, Latest_webUIVersion string, " + + "Latest_webUITypeCarrVer string, Latest_webTypeDataVerNumber string, " + + "Latest_operatorsVersion string, Latest_phonePADPartitionedVersions string, " + + "Latest_operatorId string, gamePointDescription string,gamePointId double,contractNumber " + + "BigInt) STORED BY 'org.apache.carbondata.format'") + sql( + s"LOAD DATA INPATH '$resourcesPath/100_olap.csv' INTO table TCarbonSource options " + + "('DELIMITER'=',', 'QUOTECHAR'='\', 'FILEHEADER'='imei,deviceInformationId,MAC,deviceColor," + + "device_backColor,modelId,marketName,AMSize,ROMSize,CUPAudit,CPIClocked,series," + + "productionDate,bomCode,internalModels,deliveryTime,channelsId,channelsName,deliveryAreaId," + + "deliveryCountry,deliveryProvince,deliveryCity,deliveryDistrict,deliveryStreet," + + "oxSingleNumber,ActiveCheckTime,ActiveAreaId,ActiveCountry,ActiveProvince,Activecity," + + "ActiveDistrict,ActiveStreet,ActiveOperatorId,Active_releaseId,Active_EMUIVersion," + + "Active_operaSysVersion,Active_BacVerNumber,Active_BacFlashVer,Active_webUIVersion," + + "Active_webUITypeCarrVer,Active_webTypeDataVerNumber,Active_operatorsVersion," + + "Active_phonePADPartitionedVersions,Latest_YEAR,Latest_MONTH,Latest_DAY,Latest_HOUR," + + "Latest_areaId,Latest_country,Latest_province,Latest_city,Latest_district,Latest_street," + + "Latest_releaseId,Latest_EMUIVersion,Latest_operaSysVersion,Latest_BacVerNumber," + + "Latest_BacFlashVer,Latest_webUIVersion,Latest_webUITypeCarrVer," + + "Latest_webTypeDataVerNumber,Latest_operatorsVersion,Latest_phonePADPartitionedVersions," + + "Latest_operatorId,gamePointDescription,gamePointId,contractNumber', " + + "'bad_records_logger_enable'='false','bad_records_action'='FORCE')") + } + + test("insert into hive") { + sql("drop table if exists thive2") + sql( + "create table thive2 row format delimited fields terminated by '\017' stored as textfile as" + + " " + --- End diff -- remove this, and format the statement into proper line wrap. ``` CREATE TABLE xxx ROW FORMAT DELIMITED ... STORED AS ... AS SELECT ... ``` --- |
In reply to this post by qiuchenjian-2
Github user ManoharVanam commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1683#discussion_r158196677 --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/insertQuery/InsertIntoNonCarbonTableTestCase.scala --- @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.carbondata.spark.testsuite.insertQuery + +import org.apache.spark.sql.Row +import org.apache.spark.sql.test.util.QueryTest +import org.scalatest.BeforeAndAfterAll + + +class InsertIntoNonCarbonTableTestCase extends QueryTest with BeforeAndAfterAll { + override def beforeAll { + sql("drop table if exists TCarbonSource") + sql( + "create table TCarbonSource (imei string,deviceInformationId int,MAC string,deviceColor " + + "string,device_backColor string,modelId string,marketName string,AMSize string,ROMSize " + + "string,CUPAudit string,CPIClocked string,series string,productionDate timestamp,bomCode " + + "string,internalModels string, deliveryTime string, channelsId string, channelsName string " + + ", deliveryAreaId string, deliveryCountry string, deliveryProvince string, deliveryCity " + + "string,deliveryDistrict string, deliveryStreet string, oxSingleNumber string, " + + "ActiveCheckTime string, ActiveAreaId string, ActiveCountry string, ActiveProvince string, " + + "Activecity string, ActiveDistrict string, ActiveStreet string, ActiveOperatorId string, " + + "Active_releaseId string, Active_EMUIVersion string, Active_operaSysVersion string, " + + "Active_BacVerNumber string, Active_BacFlashVer string, Active_webUIVersion string, " + + "Active_webUITypeCarrVer string,Active_webTypeDataVerNumber string, Active_operatorsVersion" + + " string, Active_phonePADPartitionedVersions string, Latest_YEAR int, Latest_MONTH int, " + + "Latest_DAY Decimal(30,10), Latest_HOUR string, Latest_areaId string, Latest_country " + + "string, Latest_province string, Latest_city string, Latest_district string, Latest_street " + + "string, Latest_releaseId string, Latest_EMUIVersion string, Latest_operaSysVersion string," + + " Latest_BacVerNumber string, Latest_BacFlashVer string, Latest_webUIVersion string, " + + "Latest_webUITypeCarrVer string, Latest_webTypeDataVerNumber string, " + + "Latest_operatorsVersion string, Latest_phonePADPartitionedVersions string, " + + "Latest_operatorId string, gamePointDescription string,gamePointId double,contractNumber " + + "BigInt) STORED BY 'org.apache.carbondata.format'") + sql( + s"LOAD DATA INPATH '$resourcesPath/100_olap.csv' INTO table TCarbonSource options " + + "('DELIMITER'=',', 'QUOTECHAR'='\', 'FILEHEADER'='imei,deviceInformationId,MAC,deviceColor," + + "device_backColor,modelId,marketName,AMSize,ROMSize,CUPAudit,CPIClocked,series," + + "productionDate,bomCode,internalModels,deliveryTime,channelsId,channelsName,deliveryAreaId," + + "deliveryCountry,deliveryProvince,deliveryCity,deliveryDistrict,deliveryStreet," + + "oxSingleNumber,ActiveCheckTime,ActiveAreaId,ActiveCountry,ActiveProvince,Activecity," + + "ActiveDistrict,ActiveStreet,ActiveOperatorId,Active_releaseId,Active_EMUIVersion," + + "Active_operaSysVersion,Active_BacVerNumber,Active_BacFlashVer,Active_webUIVersion," + + "Active_webUITypeCarrVer,Active_webTypeDataVerNumber,Active_operatorsVersion," + + "Active_phonePADPartitionedVersions,Latest_YEAR,Latest_MONTH,Latest_DAY,Latest_HOUR," + + "Latest_areaId,Latest_country,Latest_province,Latest_city,Latest_district,Latest_street," + + "Latest_releaseId,Latest_EMUIVersion,Latest_operaSysVersion,Latest_BacVerNumber," + + "Latest_BacFlashVer,Latest_webUIVersion,Latest_webUITypeCarrVer," + + "Latest_webTypeDataVerNumber,Latest_operatorsVersion,Latest_phonePADPartitionedVersions," + + "Latest_operatorId,gamePointDescription,gamePointId,contractNumber', " + + "'bad_records_logger_enable'='false','bad_records_action'='FORCE')") + } + + test("insert into hive") { + sql("drop table if exists thive2") + sql( + "create table thive2 row format delimited fields terminated by '\017' stored as textfile as" + + " " + --- End diff -- @jackylk is there any option to add these rules in scala style? --- |
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1683#discussion_r158416107 --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/insertQuery/InsertIntoNonCarbonTableTestCase.scala --- @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.carbondata.spark.testsuite.insertQuery + +import org.apache.spark.sql.Row +import org.apache.spark.sql.test.util.QueryTest +import org.scalatest.BeforeAndAfterAll + + +class InsertIntoNonCarbonTableTestCase extends QueryTest with BeforeAndAfterAll { + override def beforeAll { + sql("drop table if exists TCarbonSource") + sql( + "create table TCarbonSource (imei string,deviceInformationId int,MAC string,deviceColor " + + "string,device_backColor string,modelId string,marketName string,AMSize string,ROMSize " + + "string,CUPAudit string,CPIClocked string,series string,productionDate timestamp,bomCode " + + "string,internalModels string, deliveryTime string, channelsId string, channelsName string " + + ", deliveryAreaId string, deliveryCountry string, deliveryProvince string, deliveryCity " + + "string,deliveryDistrict string, deliveryStreet string, oxSingleNumber string, " + + "ActiveCheckTime string, ActiveAreaId string, ActiveCountry string, ActiveProvince string, " + + "Activecity string, ActiveDistrict string, ActiveStreet string, ActiveOperatorId string, " + + "Active_releaseId string, Active_EMUIVersion string, Active_operaSysVersion string, " + + "Active_BacVerNumber string, Active_BacFlashVer string, Active_webUIVersion string, " + + "Active_webUITypeCarrVer string,Active_webTypeDataVerNumber string, Active_operatorsVersion" + + " string, Active_phonePADPartitionedVersions string, Latest_YEAR int, Latest_MONTH int, " + + "Latest_DAY Decimal(30,10), Latest_HOUR string, Latest_areaId string, Latest_country " + + "string, Latest_province string, Latest_city string, Latest_district string, Latest_street " + + "string, Latest_releaseId string, Latest_EMUIVersion string, Latest_operaSysVersion string," + + " Latest_BacVerNumber string, Latest_BacFlashVer string, Latest_webUIVersion string, " + + "Latest_webUITypeCarrVer string, Latest_webTypeDataVerNumber string, " + + "Latest_operatorsVersion string, Latest_phonePADPartitionedVersions string, " + + "Latest_operatorId string, gamePointDescription string,gamePointId double,contractNumber " + + "BigInt) STORED BY 'org.apache.carbondata.format'") + sql( + s"LOAD DATA INPATH '$resourcesPath/100_olap.csv' INTO table TCarbonSource options " + + "('DELIMITER'=',', 'QUOTECHAR'='\', 'FILEHEADER'='imei,deviceInformationId,MAC,deviceColor," + + "device_backColor,modelId,marketName,AMSize,ROMSize,CUPAudit,CPIClocked,series," + + "productionDate,bomCode,internalModels,deliveryTime,channelsId,channelsName,deliveryAreaId," + + "deliveryCountry,deliveryProvince,deliveryCity,deliveryDistrict,deliveryStreet," + + "oxSingleNumber,ActiveCheckTime,ActiveAreaId,ActiveCountry,ActiveProvince,Activecity," + + "ActiveDistrict,ActiveStreet,ActiveOperatorId,Active_releaseId,Active_EMUIVersion," + + "Active_operaSysVersion,Active_BacVerNumber,Active_BacFlashVer,Active_webUIVersion," + + "Active_webUITypeCarrVer,Active_webTypeDataVerNumber,Active_operatorsVersion," + + "Active_phonePADPartitionedVersions,Latest_YEAR,Latest_MONTH,Latest_DAY,Latest_HOUR," + + "Latest_areaId,Latest_country,Latest_province,Latest_city,Latest_district,Latest_street," + + "Latest_releaseId,Latest_EMUIVersion,Latest_operaSysVersion,Latest_BacVerNumber," + + "Latest_BacFlashVer,Latest_webUIVersion,Latest_webUITypeCarrVer," + + "Latest_webTypeDataVerNumber,Latest_operatorsVersion,Latest_phonePADPartitionedVersions," + + "Latest_operatorId,gamePointDescription,gamePointId,contractNumber', " + + "'bad_records_logger_enable'='false','bad_records_action'='FORCE')") + } + + test("insert into hive") { + sql("drop table if exists thive2") + sql( + "create table thive2 row format delimited fields terminated by '\017' stored as textfile as" + + " " + --- End diff -- I do not think there are such rules, since it is SQL specific --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1683 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2244/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1683 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1021/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1683 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2502/ --- |
In reply to this post by qiuchenjian-2
Github user ManoharVanam commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1683#discussion_r158429172 --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/insertQuery/InsertIntoNonCarbonTableTestCase.scala --- @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.carbondata.spark.testsuite.insertQuery + +import org.apache.spark.sql.Row +import org.apache.spark.sql.test.util.QueryTest +import org.scalatest.BeforeAndAfterAll + + +class InsertIntoNonCarbonTableTestCase extends QueryTest with BeforeAndAfterAll { + override def beforeAll { + sql("drop table if exists TCarbonSource") + sql( + "create table TCarbonSource (imei string,deviceInformationId int,MAC string,deviceColor " + + "string,device_backColor string,modelId string,marketName string,AMSize string,ROMSize " + + "string,CUPAudit string,CPIClocked string,series string,productionDate timestamp,bomCode " + + "string,internalModels string, deliveryTime string, channelsId string, channelsName string " + + ", deliveryAreaId string, deliveryCountry string, deliveryProvince string, deliveryCity " + + "string,deliveryDistrict string, deliveryStreet string, oxSingleNumber string, " + + "ActiveCheckTime string, ActiveAreaId string, ActiveCountry string, ActiveProvince string, " + + "Activecity string, ActiveDistrict string, ActiveStreet string, ActiveOperatorId string, " + + "Active_releaseId string, Active_EMUIVersion string, Active_operaSysVersion string, " + + "Active_BacVerNumber string, Active_BacFlashVer string, Active_webUIVersion string, " + + "Active_webUITypeCarrVer string,Active_webTypeDataVerNumber string, Active_operatorsVersion" + + " string, Active_phonePADPartitionedVersions string, Latest_YEAR int, Latest_MONTH int, " + + "Latest_DAY Decimal(30,10), Latest_HOUR string, Latest_areaId string, Latest_country " + + "string, Latest_province string, Latest_city string, Latest_district string, Latest_street " + + "string, Latest_releaseId string, Latest_EMUIVersion string, Latest_operaSysVersion string," + + " Latest_BacVerNumber string, Latest_BacFlashVer string, Latest_webUIVersion string, " + + "Latest_webUITypeCarrVer string, Latest_webTypeDataVerNumber string, " + + "Latest_operatorsVersion string, Latest_phonePADPartitionedVersions string, " + + "Latest_operatorId string, gamePointDescription string,gamePointId double,contractNumber " + + "BigInt) STORED BY 'org.apache.carbondata.format'") + sql( + s"LOAD DATA INPATH '$resourcesPath/100_olap.csv' INTO table TCarbonSource options " + + "('DELIMITER'=',', 'QUOTECHAR'='\', 'FILEHEADER'='imei,deviceInformationId,MAC,deviceColor," + + "device_backColor,modelId,marketName,AMSize,ROMSize,CUPAudit,CPIClocked,series," + + "productionDate,bomCode,internalModels,deliveryTime,channelsId,channelsName,deliveryAreaId," + + "deliveryCountry,deliveryProvince,deliveryCity,deliveryDistrict,deliveryStreet," + + "oxSingleNumber,ActiveCheckTime,ActiveAreaId,ActiveCountry,ActiveProvince,Activecity," + + "ActiveDistrict,ActiveStreet,ActiveOperatorId,Active_releaseId,Active_EMUIVersion," + + "Active_operaSysVersion,Active_BacVerNumber,Active_BacFlashVer,Active_webUIVersion," + + "Active_webUITypeCarrVer,Active_webTypeDataVerNumber,Active_operatorsVersion," + + "Active_phonePADPartitionedVersions,Latest_YEAR,Latest_MONTH,Latest_DAY,Latest_HOUR," + + "Latest_areaId,Latest_country,Latest_province,Latest_city,Latest_district,Latest_street," + + "Latest_releaseId,Latest_EMUIVersion,Latest_operaSysVersion,Latest_BacVerNumber," + + "Latest_BacFlashVer,Latest_webUIVersion,Latest_webUITypeCarrVer," + + "Latest_webTypeDataVerNumber,Latest_operatorsVersion,Latest_phonePADPartitionedVersions," + + "Latest_operatorId,gamePointDescription,gamePointId,contractNumber', " + + "'bad_records_logger_enable'='false','bad_records_action'='FORCE')") + } + + test("insert into hive") { + sql("drop table if exists thive2") + sql( + "create table thive2 row format delimited fields terminated by '\017' stored as textfile as" + + " " + --- End diff -- done --- |
In reply to this post by qiuchenjian-2
Github user ManoharVanam commented on the issue:
https://github.com/apache/carbondata/pull/1683 retest sdv please --- |
In reply to this post by qiuchenjian-2
|
In reply to this post by qiuchenjian-2
|
Free forum by Nabble | Edit this page |