[GitHub] [carbondata] vikramahuja1001 opened a new pull request #3775: [WIP] Bloom Fallback Issue

[GitBox]

vikramahuja1001 opened a new pull request #3775:
URL: https://github.com/apache/carbondata/pull/3775


    ### Why is this PR needed?
   
   
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3775:
URL: https://github.com/apache/carbondata/pull/3775#issuecomment-634316188






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

vikramahuja1001 commented on a change in pull request #3775:
URL: https://github.com/apache/carbondata/pull/3775#discussion_r432286530



##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/index/CarbonCreateIndexCommand.scala
##########
@@ -238,11 +238,22 @@ case class CarbonCreateIndexCommand(
               false)
             val enabledIndexInfo = IndexTableInfo.enableIndex(indexInfo, indexModel.indexName)
 
-            // set index information in parent table
-            val parentIndexMetadata = parentTable.getIndexMetadata
-            parentIndexMetadata.updateIndexStatus(indexProviderName,
-              indexModel.indexName,
-              IndexStatus.ENABLED.name())
+            // set index information in parent table. Create it if it is null.
+            val parentIndexMetadata = if (
+              parentTable.getTableInfo.getFactTable.getTableProperties
+                .get(parentTable.getCarbonTableIdentifier.getTableId) != null) {

Review comment:
       in the case when the whole flow goes to fallback mode in the index server, the parent table retrieved again in the processData function does not contain the serialized IndexMetaData.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3775:
URL: https://github.com/apache/carbondata/pull/3775#issuecomment-656660136


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3350/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3775:
URL: https://github.com/apache/carbondata/pull/3775#discussion_r457209918



##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/index/CarbonCreateIndexCommand.scala
##########
@@ -238,11 +238,22 @@ case class CarbonCreateIndexCommand(
               false)
             val enabledIndexInfo = IndexTableInfo.enableIndex(indexInfo, indexModel.indexName)
 
-            // set index information in parent table
-            val parentIndexMetadata = parentTable.getIndexMetadata
-            parentIndexMetadata.updateIndexStatus(indexProviderName,
-              indexModel.indexName,
-              IndexStatus.ENABLED.name())
+            // set index information in parent table. Create it if it is null.
+            val parentIndexMetadata = if (
+              parentTable.getTableInfo.getFactTable.getTableProperties
+                .get(parentTable.getCarbonTableIdentifier.getTableId) != null) {
+              val tempIndexMetaData = parentTable.getIndexMetadata
+              tempIndexMetaData.updateIndexStatus(indexProviderName,

Review comment:
       move this after the if-else block




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

vikramahuja1001 commented on a change in pull request #3775:
URL: https://github.com/apache/carbondata/pull/3775#discussion_r467689041



##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/index/CarbonCreateIndexCommand.scala
##########
@@ -238,11 +238,22 @@ case class CarbonCreateIndexCommand(
               false)
             val enabledIndexInfo = IndexTableInfo.enableIndex(indexInfo, indexModel.indexName)
 
-            // set index information in parent table
-            val parentIndexMetadata = parentTable.getIndexMetadata
-            parentIndexMetadata.updateIndexStatus(indexProviderName,
-              indexModel.indexName,
-              IndexStatus.ENABLED.name())
+            // set index information in parent table. Create it if it is null.
+            val parentIndexMetadata = if (
+              parentTable.getTableInfo.getFactTable.getTableProperties
+                .get(parentTable.getCarbonTableIdentifier.getTableId) != null) {

Review comment:
       @kevinjmh , in the case of spark-sql, the same jvm is used between driver and executor thus in fallback mode when cache is cleared in executor side, the serialized metadata information is also removed from the driver side.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

vikramahuja1001 commented on a change in pull request #3775:
URL: https://github.com/apache/carbondata/pull/3775#discussion_r467692421



##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/index/CarbonCreateIndexCommand.scala
##########
@@ -238,11 +238,22 @@ case class CarbonCreateIndexCommand(
               false)
             val enabledIndexInfo = IndexTableInfo.enableIndex(indexInfo, indexModel.indexName)
 
-            // set index information in parent table
-            val parentIndexMetadata = parentTable.getIndexMetadata
-            parentIndexMetadata.updateIndexStatus(indexProviderName,
-              indexModel.indexName,
-              IndexStatus.ENABLED.name())
+            // set index information in parent table. Create it if it is null.
+            val parentIndexMetadata = if (
+              parentTable.getTableInfo.getFactTable.getTableProperties
+                .get(parentTable.getCarbonTableIdentifier.getTableId) != null) {
+              val tempIndexMetaData = parentTable.getIndexMetadata
+              tempIndexMetaData.updateIndexStatus(indexProviderName,

Review comment:
       done, please check again!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3775:
URL: https://github.com/apache/carbondata/pull/3775#issuecomment-671211837


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3670/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3775:
URL: https://github.com/apache/carbondata/pull/3775#issuecomment-671213118


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1931/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on pull request #3775:
URL: https://github.com/apache/carbondata/pull/3775#issuecomment-671387564


   LGTM


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

asfgit closed pull request #3775:
URL: https://github.com/apache/carbondata/pull/3775


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]