[GitHub] [carbondata] vikramahuja1001 opened a new pull request #3565: Changes to show metacache command

[GitBox]

vikramahuja1001 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#issuecomment-573515090
 
 
   retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#issuecomment-573532037
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1615/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r367227011
 
 

 ##########
 File path: integration/spark-carbon-common-test/src/test/scala/org/apache/carbondata/sql/commands/TestCarbonShowCacheCommand.scala
 ##########
 @@ -175,13 +175,13 @@ class TestCarbonShowCacheCommand extends QueryTest with BeforeAndAfterAll {
     // Empty database
     sql("use cache_empty_db").collect()
     val result1 = sql("show metacache").collect()
-    assertResult(2)(result1.length)
-    assertResult(Row("cache_empty_db", "ALL", "0 B", "0 B", "0 B", "DRIVER"))(result1(1))
+    sql("show metacache").show()
 
 Review comment:
   Remove this

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r367232302
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
 ##########
 @@ -87,7 +108,8 @@ case class CarbonShowCacheCommand(tableIdentifier: Option[TableIdentifier],
       val indexRawResults = if (CarbonProperties.getInstance().isDistributedPruningEnabled
       (tableIdentifier.get.database.getOrElse(sparkSession.catalog.currentDatabase),
         tableIdentifier.get.table)) {
-        getTableCacheFromIndexServer(carbonTable, numberOfIndexFiles)(sparkSession)
+        getTableCacheFromIndexServer(carbonTable,
+          numberOfIndexFiles)(showExecutorCache)(sparkSession)
 
 Review comment:
   Revert this change

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r367232430
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
 ##########
 @@ -110,13 +132,41 @@ case class CarbonShowCacheCommand(tableIdentifier: Option[TableIdentifier],
     }
   }
 
+  def getAllExecutorCache(sparkSession: SparkSession): Seq[Row] = {
+    val isDistributedPruningEnabled = CarbonProperties.getInstance()
+      .isDistributedPruningEnabled("", "")
+    if (!isDistributedPruningEnabled) {
+      // Block here. this feature is only with index server enabled
+      throw new UnsupportedOperationException(
+        "Show Executor Metacache is only avalable with Index Server Enabled")
+      Seq()
 
 Review comment:
   no need to return Seq() if you are already throwing exception

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r367232863
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
 ##########
 @@ -169,41 +224,32 @@ case class CarbonShowCacheCommand(tableIdentifier: Option[TableIdentifier],
       }
       val (driverIndexSize, driverDatamapSize, allDictSize) = getAllDriverCacheSize(tablePaths
         .toList)
-      if (driverRows.nonEmpty) {
+      if (driverIndexSize + driverDatamapSize + allDictSize != 0 && driverRows.size!=0) {
         (Seq(
-          Row("ALL", "ALL", driverIndexSize, driverDatamapSize, allDictSize, "DRIVER"),
-          Row(currentDatabase,
-            "ALL",
-            driverdbIndexSize,
-            driverdbDatamapSize,
-            driverdbDictSize,
-            "DRIVER")
+          Row("TOTAL", driverIndexSize, driverDatamapSize, allDictSize, "DRIVER")
         ) ++ driverRows).collect {
-          case row if row.getLong(2) != 0L || row.getLong(3) != 0L || row.getLong(4) != 0L =>
-            Row(row(0), row(1), bytesToDisplaySize(row.getLong(2)),
-              bytesToDisplaySize(row.getLong(3)), bytesToDisplaySize(row.getLong(4)), "DRIVER")
+          case row if row.getLong(1) != 0L || row.getLong(2) != 0L || row.getLong(3) != 0L =>
+            Row(row(0), bytesToDisplaySize(row.getLong(1)),
+              bytesToDisplaySize(row.getLong(2)), bytesToDisplaySize(row.getLong(3)), "DRIVER")
         }
       } else {
-        makeEmptyCacheRows(currentDatabase)
+        Seq()
       }
     } else {
-      makeEmptyCacheRows(currentDatabase)
+      Seq()
     }
 
-    //      val (serverIndexSize, serverDataMapSize) = getAllIndexServerCacheSize
-    val indexDisplayRows = if (indexServerRows.nonEmpty) {
+    val indexDisplayRows = if (indexAllIndexSize + indexAllDatamapSize != 0 &&
+                               indexServerRows.size != 0) {
       (Seq(
-        Row("ALL", "ALL", indexAllIndexSize, indexAllDatamapSize, indexAllDictSize, "INDEX SERVER"),
-        Row(currentDatabase,
-          "ALL",
-          indexdbIndexSize,
-          indexdbDatamapSize,
-          driverdbDictSize,
-          "INDEX SERVER")
+        Row("TOTAL", indexAllIndexSize, indexAllDatamapSize, indexAllDictSize, "INDEX SERVER")
       ) ++ indexServerRows).collect {
-        case row if row.getLong(2) != 0L || row.getLong(3) != 0L || row.getLong(4) != 0L =>
-          Row(row.get(0), row.get(1), bytesToDisplaySize(row.getLong(2)),
-            bytesToDisplaySize(row.getLong(3)), bytesToDisplaySize(row.getLong(4)), "INDEX SERVER")
+        case row if row.getLong(1) != 0L || row.getLong(2) != 0L || row.getLong(3) != 0L =>
+          Row(row.get(0),
+            bytesToDisplaySize(row.getLong(1)),
+            bytesToDisplaySize(row.getLong(2)),
 
 Review comment:
   revert this

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r367233023
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala
 ##########
 @@ -560,9 +560,9 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser {
     }
 
   protected lazy val showCache: Parser[LogicalPlan] =
-    SHOW ~> METACACHE ~> opt(ontable) <~ opt(";") ^^ {
-      case table =>
-        CarbonShowCacheCommand(table)
+    (SHOW ~> opt(EXECUTOR) <~ METACACHE) ~ opt(ontable) <~ opt(";") ^^ {
 
 Review comment:
   add this DDL change in documentation

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r367232302
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
 ##########
 @@ -87,7 +108,8 @@ case class CarbonShowCacheCommand(tableIdentifier: Option[TableIdentifier],
       val indexRawResults = if (CarbonProperties.getInstance().isDistributedPruningEnabled
       (tableIdentifier.get.database.getOrElse(sparkSession.catalog.currentDatabase),
         tableIdentifier.get.table)) {
-        getTableCacheFromIndexServer(carbonTable, numberOfIndexFiles)(sparkSession)
+        getTableCacheFromIndexServer(carbonTable,
+          numberOfIndexFiles)(showExecutorCache)(sparkSession)
 
 Review comment:
   Revert this change

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#issuecomment-575003820
 
 
   retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r367249750
 
 

 ##########
 File path: integration/spark-carbon-common-test/src/test/scala/org/apache/carbondata/sql/commands/TestCarbonShowCacheCommand.scala
 ##########
 @@ -175,13 +175,13 @@ class TestCarbonShowCacheCommand extends QueryTest with BeforeAndAfterAll {
     // Empty database
     sql("use cache_empty_db").collect()
     val result1 = sql("show metacache").collect()
-    assertResult(2)(result1.length)
-    assertResult(Row("cache_empty_db", "ALL", "0 B", "0 B", "0 B", "DRIVER"))(result1(1))
+    sql("show metacache").show()
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r367249765
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
 ##########
 @@ -110,13 +132,41 @@ case class CarbonShowCacheCommand(tableIdentifier: Option[TableIdentifier],
     }
   }
 
+  def getAllExecutorCache(sparkSession: SparkSession): Seq[Row] = {
+    val isDistributedPruningEnabled = CarbonProperties.getInstance()
+      .isDistributedPruningEnabled("", "")
+    if (!isDistributedPruningEnabled) {
+      // Block here. this feature is only with index server enabled
+      throw new UnsupportedOperationException(
+        "Show Executor Metacache is only avalable with Index Server Enabled")
+      Seq()
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r367249802
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
 ##########
 @@ -169,41 +224,32 @@ case class CarbonShowCacheCommand(tableIdentifier: Option[TableIdentifier],
       }
       val (driverIndexSize, driverDatamapSize, allDictSize) = getAllDriverCacheSize(tablePaths
         .toList)
-      if (driverRows.nonEmpty) {
+      if (driverIndexSize + driverDatamapSize + allDictSize != 0 && driverRows.size!=0) {
         (Seq(
-          Row("ALL", "ALL", driverIndexSize, driverDatamapSize, allDictSize, "DRIVER"),
-          Row(currentDatabase,
-            "ALL",
-            driverdbIndexSize,
-            driverdbDatamapSize,
-            driverdbDictSize,
-            "DRIVER")
+          Row("TOTAL", driverIndexSize, driverDatamapSize, allDictSize, "DRIVER")
         ) ++ driverRows).collect {
-          case row if row.getLong(2) != 0L || row.getLong(3) != 0L || row.getLong(4) != 0L =>
-            Row(row(0), row(1), bytesToDisplaySize(row.getLong(2)),
-              bytesToDisplaySize(row.getLong(3)), bytesToDisplaySize(row.getLong(4)), "DRIVER")
+          case row if row.getLong(1) != 0L || row.getLong(2) != 0L || row.getLong(3) != 0L =>
+            Row(row(0), bytesToDisplaySize(row.getLong(1)),
+              bytesToDisplaySize(row.getLong(2)), bytesToDisplaySize(row.getLong(3)), "DRIVER")
         }
       } else {
-        makeEmptyCacheRows(currentDatabase)
+        Seq()
       }
     } else {
-      makeEmptyCacheRows(currentDatabase)
+      Seq()
     }
 
-    //      val (serverIndexSize, serverDataMapSize) = getAllIndexServerCacheSize
-    val indexDisplayRows = if (indexServerRows.nonEmpty) {
+    val indexDisplayRows = if (indexAllIndexSize + indexAllDatamapSize != 0 &&
+                               indexServerRows.size != 0) {
       (Seq(
-        Row("ALL", "ALL", indexAllIndexSize, indexAllDatamapSize, indexAllDictSize, "INDEX SERVER"),
-        Row(currentDatabase,
-          "ALL",
-          indexdbIndexSize,
-          indexdbDatamapSize,
-          driverdbDictSize,
-          "INDEX SERVER")
+        Row("TOTAL", indexAllIndexSize, indexAllDatamapSize, indexAllDictSize, "INDEX SERVER")
       ) ++ indexServerRows).collect {
-        case row if row.getLong(2) != 0L || row.getLong(3) != 0L || row.getLong(4) != 0L =>
-          Row(row.get(0), row.get(1), bytesToDisplaySize(row.getLong(2)),
-            bytesToDisplaySize(row.getLong(3)), bytesToDisplaySize(row.getLong(4)), "INDEX SERVER")
+        case row if row.getLong(1) != 0L || row.getLong(2) != 0L || row.getLong(3) != 0L =>
+          Row(row.get(0),
+            bytesToDisplaySize(row.getLong(1)),
+            bytesToDisplaySize(row.getLong(2)),
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r367249817
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala
 ##########
 @@ -560,9 +560,9 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser {
     }
 
   protected lazy val showCache: Parser[LogicalPlan] =
-    SHOW ~> METACACHE ~> opt(ontable) <~ opt(";") ^^ {
-      case table =>
-        CarbonShowCacheCommand(table)
+    (SHOW ~> opt(EXECUTOR) <~ METACACHE) ~ opt(ontable) <~ opt(";") ^^ {
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#issuecomment-575005149
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1653/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#issuecomment-575024665
 
 
   retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#issuecomment-575026364
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1655/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#issuecomment-575121589
 
 
   retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#issuecomment-575150302
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1667/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#issuecomment-575150865
 
 
   @kunal642 , done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#issuecomment-575557523
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1677/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services