[GitHub] [carbondata] ShreelekhyaG opened a new pull request #4147: [WIP] Fix issue when refresh main table with MV

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] ShreelekhyaG opened a new pull request #4147: [WIP] Fix issue when refresh main table with MV

GitBox

ShreelekhyaG opened a new pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147


    ### Why is this PR needed?
    When trying to register a table of old store which has MV, it fails parser error(syntax issue while creating table). It is trying to create table with `relatedmvtablesmap` property which is not valid.
   
    ### What changes were proposed in this PR?
   - Removed `relatedmvtablesmap` from table properties in `RefreshCarbonTableCommand`
   - After Main table has registered, to register MV made changes to get the schema from the system folder and register.
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - Yes
   
       
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [WIP] Fix issue when refresh main table with MV

GitBox

CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-854858593


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3763/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [WIP] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-854902879


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5506/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] ShreelekhyaG commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

ShreelekhyaG commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856489014


   retest this please


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] Indhumathi27 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

Indhumathi27 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856514647


   LGTM


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856560360


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5519/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856563176


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3776/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

Indhumathi27 commented on a change in pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#discussion_r647299409



##########
File path: integration/spark/src/test/scala/org/apache/spark/carbondata/register/TestRegisterCarbonTable.scala
##########
@@ -231,6 +232,45 @@ class TestRegisterCarbonTable extends QueryTest with BeforeAndAfterEach {
     }
   }
 
+  test("test register table with mv") {
+    sql(s"create database carbon location '$dbLocation'")
+    sql("use carbon")
+    sql(
+      """create table carbon.carbontable (
+        |c1 string,c2 int,c3 string,c5 string) STORED AS carbondata""".stripMargin)
+    sql("insert into carbontable select 'a',1,'aa','aaa'")
+    sql("create materialized view mv1 as select avg(c2),c3 from carbontable group by c3")
+    backUpData(dbLocation, Some("carbon"), "carbontable")
+    backUpData(dbLocation, Some("carbon"), "mv1")
+    val source = dbLocation + CarbonCommonConstants.FILE_SEPARATOR + "_system" +
+                 CarbonCommonConstants.FILE_SEPARATOR + "mv_schema.mv1"
+    val destination = dbLocation + "_back" + CarbonCommonConstants.FILE_SEPARATOR + "mv_schema.mv1"
+    backUpMvSchema(source, destination)
+    sql("drop table carbontable")
+    if (!CarbonEnv.getInstance(sqlContext.sparkSession).carbonMetaStore.isReadFromHiveMetaStore) {
+      restoreData(dbLocation, "carbontable")
+      restoreData(dbLocation, "mv1")
+      backUpMvSchema(destination, source)
+      sql("refresh table carbontable")
+      sql("refresh table mv1")
+      sql("refresh materialized view mv1")
+      checkAnswer(sql("select count(*) from carbontable"), Row(1))
+      checkAnswer(sql("select c1 from carbontable"), Seq(Row("a")))
+      sql("show materialized views on table carbontable").show(false)

Review comment:
       please remove this show




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856654750


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5522/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856656830


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3779/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] ShreelekhyaG commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

ShreelekhyaG commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856665109


   retest this please


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856726384


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3781/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] ShreelekhyaG commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

ShreelekhyaG commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856745833


   retest this please


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856864464


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3783/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856867436


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5526/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

ShreelekhyaG commented on a change in pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#discussion_r647969173



##########
File path: integration/spark/src/test/scala/org/apache/spark/carbondata/register/TestRegisterCarbonTable.scala
##########
@@ -231,6 +232,45 @@ class TestRegisterCarbonTable extends QueryTest with BeforeAndAfterEach {
     }
   }
 
+  test("test register table with mv") {
+    sql(s"create database carbon location '$dbLocation'")
+    sql("use carbon")
+    sql(
+      """create table carbon.carbontable (
+        |c1 string,c2 int,c3 string,c5 string) STORED AS carbondata""".stripMargin)
+    sql("insert into carbontable select 'a',1,'aa','aaa'")
+    sql("create materialized view mv1 as select avg(c2),c3 from carbontable group by c3")
+    backUpData(dbLocation, Some("carbon"), "carbontable")
+    backUpData(dbLocation, Some("carbon"), "mv1")
+    val source = dbLocation + CarbonCommonConstants.FILE_SEPARATOR + "_system" +
+                 CarbonCommonConstants.FILE_SEPARATOR + "mv_schema.mv1"
+    val destination = dbLocation + "_back" + CarbonCommonConstants.FILE_SEPARATOR + "mv_schema.mv1"
+    backUpMvSchema(source, destination)
+    sql("drop table carbontable")
+    if (!CarbonEnv.getInstance(sqlContext.sparkSession).carbonMetaStore.isReadFromHiveMetaStore) {
+      restoreData(dbLocation, "carbontable")
+      restoreData(dbLocation, "mv1")
+      backUpMvSchema(destination, source)
+      sql("refresh table carbontable")
+      sql("refresh table mv1")
+      sql("refresh materialized view mv1")
+      checkAnswer(sql("select count(*) from carbontable"), Row(1))
+      checkAnswer(sql("select c1 from carbontable"), Seq(Row("a")))
+      sql("show materialized views on table carbontable").show(false)

Review comment:
       Done




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] Indhumathi27 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

Indhumathi27 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-858311083


   retest this please


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-858387425


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3792/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-858404376


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5535/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] asfgit closed pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV

GitBox
In reply to this post by GitBox

asfgit closed pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147


   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]