[GitHub] [carbondata] vikramahuja1001 opened a new pull request #3537: Metacache issues

[GitBox]

kunal642 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362374380
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
 ##########
 @@ -125,6 +127,19 @@ class CarbonEnv {
     LOGGER.info("Initialize CarbonEnv completed...")
   }
 
+
+  def createIndexServerTmp(sparkSession: SparkSession): Unit = {
 
 Review comment:
   We already have a method to create temp folder in CarbonUtil. Please use the same method and add the code to change the permission to 777

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362374604
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
 ##########
 @@ -3321,7 +3321,7 @@ public static String getIndexServerTempPath(String tablePath, String queryId) {
         .getProperty(CarbonCommonConstants.CARBON_INDEX_SERVER_TEMP_PATH);
     if (null == tempFolderPath) {
       tempFolderPath =
-          tablePath + "/" + CarbonCommonConstants.INDEX_SERVER_TEMP_FOLDER_NAME + "/" + queryId;
+          "/tmp/" + CarbonCommonConstants.INDEX_SERVER_TEMP_FOLDER_NAME + "/" + queryId;
 
 Review comment:
   use CarbonCommonConstants.FILE_SEPARATOR instead on "/" in the method

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362376155
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
 ##########
 @@ -71,6 +72,7 @@ class CarbonEnv {
       properties.addProperty(CarbonCommonConstants.STORE_LOCATION, storePath)
     }
     LOGGER.info(s"Initializing CarbonEnv, store location: $storePath")
+    createIndexServerTmp(sparkSession);
 
 Review comment:
   please add a comment, why is this required and what is this used for?
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362376184
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
 ##########
 @@ -125,6 +127,19 @@ class CarbonEnv {
     LOGGER.info("Initialize CarbonEnv completed...")
   }
 
+
 
 Review comment:
   please add method level comments

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362376259
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
 ##########
 @@ -125,6 +127,19 @@ class CarbonEnv {
     LOGGER.info("Initialize CarbonEnv completed...")
   }
 
+
+  def createIndexServerTmp(sparkSession: SparkSession): Unit = {
+    val path = CarbonUtil.checkAndAppendFileSystemURIScheme(
+      "/tmp/" + CarbonCommonConstants.INDEX_SERVER_TEMP_FOLDER_NAME)
+    if (!FileFactory.isFileExist(path)) {
+        // Create the new directory
+        LOGGER.info(s"Creating Index Server Tmp folder: ${path} ")
 
 Review comment:
   either give `tmp` or `temp` in log, remove Tmp

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362376318
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonDropCacheCommand.scala
 ##########
 @@ -49,18 +49,19 @@ case class CarbonDropCacheCommand(tableIdentifier: TableIdentifier, internalCall
     OperationListenerBus.getInstance.fireEvent(dropCacheEvent, operationContext)
 
     val cache = CacheProvider.getInstance().getCarbonCache
+    // Clearing cache from IndexServer
 
 Review comment:
   change commnet from `clearing` to `clear`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362376348
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonDropCacheCommand.scala
 ##########
 @@ -49,18 +49,19 @@ case class CarbonDropCacheCommand(tableIdentifier: TableIdentifier, internalCall
     OperationListenerBus.getInstance.fireEvent(dropCacheEvent, operationContext)
 
     val cache = CacheProvider.getInstance().getCarbonCache
+    // Clearing cache from IndexServer
+    if (CarbonProperties.getInstance().isDistributedPruningEnabled(carbonTable.getDatabaseName,
+      carbonTable.getTableName)) {
+      LOGGER.info("Clearing cache from IndexServer")
+      DataMapUtil.executeClearDataMapJob(carbonTable, DataMapUtil.DISTRIBUTED_JOB_NAME)
+    }
     if (cache != null) {
-      // Get all Index files for the specified table.
-      if (CarbonProperties.getInstance().isDistributedPruningEnabled(carbonTable.getDatabaseName,
-        carbonTable.getTableName)) {
-        DataMapUtil.executeClearDataMapJob(carbonTable, DataMapUtil.DISTRIBUTED_JOB_NAME)
-      } else {
-        // Extract dictionary keys for the table and create cache keys from those
-        val dictKeys: List[String] = CacheUtil.getAllDictCacheKeys(carbonTable)
-        // Remove elements from cache
-        cache.removeAll(dictKeys.asJava)
-        DataMapStoreManager.getInstance().clearDataMaps(carbonTable.getAbsoluteTableIdentifier)
-      }
+      LOGGER.info("Clearing cache from driver side")
+      // Extract dictionary keys for the table and create cache keys from those
 
 Review comment:
   please corect the comment, its not clear

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362406661
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
 ##########
 @@ -125,6 +127,19 @@ class CarbonEnv {
     LOGGER.info("Initialize CarbonEnv completed...")
   }
 
+
+  def createIndexServerTmp(sparkSession: SparkSession): Unit = {
+    val path = CarbonUtil.checkAndAppendFileSystemURIScheme(
+      "/tmp/" + CarbonCommonConstants.INDEX_SERVER_TEMP_FOLDER_NAME)
+    if (!FileFactory.isFileExist(path)) {
+        // Create the new directory
+        LOGGER.info(s"Creating Index Server Tmp folder: ${path} ")
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362406682
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
 ##########
 @@ -125,6 +127,19 @@ class CarbonEnv {
     LOGGER.info("Initialize CarbonEnv completed...")
   }
 
+
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362406698
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
 ##########
 @@ -71,6 +72,7 @@ class CarbonEnv {
       properties.addProperty(CarbonCommonConstants.STORE_LOCATION, storePath)
     }
     LOGGER.info(s"Initializing CarbonEnv, store location: $storePath")
+    createIndexServerTmp(sparkSession);
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362406718
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
 ##########
 @@ -3321,7 +3321,7 @@ public static String getIndexServerTempPath(String tablePath, String queryId) {
         .getProperty(CarbonCommonConstants.CARBON_INDEX_SERVER_TEMP_PATH);
     if (null == tempFolderPath) {
       tempFolderPath =
-          tablePath + "/" + CarbonCommonConstants.INDEX_SERVER_TEMP_FOLDER_NAME + "/" + queryId;
+          "/tmp/" + CarbonCommonConstants.INDEX_SERVER_TEMP_FOLDER_NAME + "/" + queryId;
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362406737
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
 ##########
 @@ -125,6 +127,19 @@ class CarbonEnv {
     LOGGER.info("Initialize CarbonEnv completed...")
   }
 
+
+  def createIndexServerTmp(sparkSession: SparkSession): Unit = {
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#issuecomment-570153160
 
 
   Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1404/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#issuecomment-570153358
 
 
   Build Failed  with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1394/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#issuecomment-570153985
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1416/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on issue #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#issuecomment-570154451
 
 
   retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3537: [CARBONDATA-3646] [CARBONDATA-3647]: 1. Select query * fails when using index server. 2. Filter query failure with Index server when loaded with global_sort_partition = 100000, giving Null pointer exception
URL: https://github.com/apache/carbondata/pull/3537#issuecomment-570159452
 
 
   Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1396/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: Fix query failure with Index Server
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362422396
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
 ##########
 @@ -71,6 +70,8 @@ class CarbonEnv {
       properties.addProperty(CarbonCommonConstants.STORE_LOCATION, storePath)
     }
     LOGGER.info(s"Initializing CarbonEnv, store location: $storePath")
+    // Creating the index server tepm folder where splits for select query is written
 
 Review comment:
   temp spelling is wrong

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: Fix query failure with Index Server
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362427225
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
 ##########
 @@ -71,6 +70,8 @@ class CarbonEnv {
       properties.addProperty(CarbonCommonConstants.STORE_LOCATION, storePath)
     }
     LOGGER.info(s"Initializing CarbonEnv, store location: $storePath")
+    // Creating the index server tepm folder where splits for select query is written
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on a change in pull request #3537: [CARBONDATA-3646] [CARBONDATA-3647]: Fix query failure with Index Server
URL: https://github.com/apache/carbondata/pull/3537#discussion_r362427255
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonDropCacheCommand.scala
 ##########
 @@ -49,18 +49,19 @@ case class CarbonDropCacheCommand(tableIdentifier: TableIdentifier, internalCall
     OperationListenerBus.getInstance.fireEvent(dropCacheEvent, operationContext)
 
     val cache = CacheProvider.getInstance().getCarbonCache
+    // Clearing cache from IndexServer
+    if (CarbonProperties.getInstance().isDistributedPruningEnabled(carbonTable.getDatabaseName,
+      carbonTable.getTableName)) {
+      LOGGER.info("Clearing cache from IndexServer")
+      DataMapUtil.executeClearDataMapJob(carbonTable, DataMapUtil.DISTRIBUTED_JOB_NAME)
+    }
     if (cache != null) {
-      // Get all Index files for the specified table.
-      if (CarbonProperties.getInstance().isDistributedPruningEnabled(carbonTable.getDatabaseName,
-        carbonTable.getTableName)) {
-        DataMapUtil.executeClearDataMapJob(carbonTable, DataMapUtil.DISTRIBUTED_JOB_NAME)
-      } else {
-        // Extract dictionary keys for the table and create cache keys from those
-        val dictKeys: List[String] = CacheUtil.getAllDictCacheKeys(carbonTable)
-        // Remove elements from cache
-        cache.removeAll(dictKeys.asJava)
-        DataMapStoreManager.getInstance().clearDataMaps(carbonTable.getAbsoluteTableIdentifier)
-      }
+      LOGGER.info("Clearing cache from driver side")
+      // Extract dictionary keys for the table and create cache keys from those
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services