[GitHub] [carbondata] ShreelekhyaG opened a new pull request #3890: [WIP] After reset query not hitting MV

[GitBox]

ShreelekhyaG opened a new pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890


    ### Why is this PR needed?
    After reset query not hitting MV.
    With the `reset`, `spark.sql.warehouse.dir` and `carbonStorePath` don't match and the databaseLocation will change to old table path format and refers to wrong mv_status file.
   
    ### What changes were proposed in this PR?
   Made changes to get the mv status file path from `SchemaProvider` so it can return the path if it is already stored in the map.
       
    ### Does this PR introduce any user interface change?
    - No
   
    ### Is any new testcase added?
    - Yes
   
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#issuecomment-673590190


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1973/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#issuecomment-673590822


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3712/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#issuecomment-674175326


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3730/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#issuecomment-674181256


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1991/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Indhumathi27 commented on a change in pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#discussion_r476187983



##########
File path: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/SetParameterTestCase.scala
##########
@@ -252,6 +252,27 @@ class SetParameterTestCase extends QueryTest with BeforeAndAfterAll {
     sql("RESET")
   }
 
+  test("TC_014-test mv after reset properties") {
+    sql("drop table if exists maintable")
+    sql("drop MATERIALIZED VIEW if exists mv1")
+    sql("CREATE TABLE maintable(empno int,empname string,projectcode int, projectjoindate " +
+        "Timestamp, projectenddate date,salary double) STORED AS carbondata")
+    sql("CREATE MATERIALIZED VIEW mv1 as select timeseries(projectenddate,'day'), sum" +
+        "(projectcode) from maintable group by timeseries(projectenddate,'day')")
+    sql("insert into maintable select 1000,'PURUJIT',00012,'2015-07-26 12:07:28','2016-05-20'," +
+        "15000.00")
+    sql("insert into maintable select 1001,'PANKAJ',00010,'2015-07-26 17:32:20','2016-05-20'," +
+        "25000.00")
+    sql("set carbon.input.segments.defualt.maintable=1")
+    checkExistence(sql("EXPLAIN select timeseries(projectenddate,'day'), sum(projectcode) from " +

Review comment:
       Can use 'TestUtil.verifyMVHit' method to verify if query hits mv or not




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Indhumathi27 commented on a change in pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#discussion_r476188040



##########
File path: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/SetParameterTestCase.scala
##########
@@ -252,6 +252,27 @@ class SetParameterTestCase extends QueryTest with BeforeAndAfterAll {
     sql("RESET")
   }
 
+  test("TC_014-test mv after reset properties") {
+    sql("drop table if exists maintable")
+    sql("drop MATERIALIZED VIEW if exists mv1")
+    sql("CREATE TABLE maintable(empno int,empname string,projectcode int, projectjoindate " +
+        "Timestamp, projectenddate date,salary double) STORED AS carbondata")
+    sql("CREATE MATERIALIZED VIEW mv1 as select timeseries(projectenddate,'day'), sum" +
+        "(projectcode) from maintable group by timeseries(projectenddate,'day')")
+    sql("insert into maintable select 1000,'PURUJIT',00012,'2015-07-26 12:07:28','2016-05-20'," +
+        "15000.00")
+    sql("insert into maintable select 1001,'PANKAJ',00010,'2015-07-26 17:32:20','2016-05-20'," +
+        "25000.00")
+    sql("set carbon.input.segments.defualt.maintable=1")
+    checkExistence(sql("EXPLAIN select timeseries(projectenddate,'day'), sum(projectcode) from " +
+                       "maintable group by timeseries(projectenddate,'day')"), true, "mv1")
+    sql("reset")
+    checkExistence(sql("EXPLAIN select timeseries(projectenddate,'day'), sum(projectcode) from " +

Review comment:
       same as above comment




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ShreelekhyaG commented on a change in pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#discussion_r478240384



##########
File path: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/SetParameterTestCase.scala
##########
@@ -252,6 +252,27 @@ class SetParameterTestCase extends QueryTest with BeforeAndAfterAll {
     sql("RESET")
   }
 
+  test("TC_014-test mv after reset properties") {
+    sql("drop table if exists maintable")
+    sql("drop MATERIALIZED VIEW if exists mv1")
+    sql("CREATE TABLE maintable(empno int,empname string,projectcode int, projectjoindate " +
+        "Timestamp, projectenddate date,salary double) STORED AS carbondata")
+    sql("CREATE MATERIALIZED VIEW mv1 as select timeseries(projectenddate,'day'), sum" +
+        "(projectcode) from maintable group by timeseries(projectenddate,'day')")
+    sql("insert into maintable select 1000,'PURUJIT',00012,'2015-07-26 12:07:28','2016-05-20'," +
+        "15000.00")
+    sql("insert into maintable select 1001,'PANKAJ',00010,'2015-07-26 17:32:20','2016-05-20'," +
+        "25000.00")
+    sql("set carbon.input.segments.defualt.maintable=1")
+    checkExistence(sql("EXPLAIN select timeseries(projectenddate,'day'), sum(projectcode) from " +

Review comment:
       ok added check with `verifyMVHit`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#issuecomment-681842224


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3885/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#issuecomment-681845603


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2144/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

akashrn5 commented on pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#issuecomment-682373676


   @ShreelekhyaG i feel, instead of fixing the impacted MV issue, we need to check why reset changes the warehouse and carbon store, because now if we just fix MV, may be later we may get the issue with other features. Please check once.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#issuecomment-683840584


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3932/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#issuecomment-683841817


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2191/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

akashrn5 commented on pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#issuecomment-691929213


   @ShreelekhyaG as discussed this is fixed in spark, as it was a spark bug, so can you please close this PR. Please provide the spark jira link in both PR and CARBONDATA-3952  jira and close both PR and jira


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ShreelekhyaG commented on pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890#issuecomment-692482869


   Closing this PR, as it is identified as spark bug. More details can be found at https://issues.apache.org/jira/browse/SPARK-31234


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ShreelekhyaG closed pull request #3890:
URL: https://github.com/apache/carbondata/pull/3890


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]