[GitHub] [carbondata] marchpure opened a new pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold

classic Classic list List threaded Threaded
32 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold

GitBox
ajantha-bhat commented on a change in pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold
URL: https://github.com/apache/carbondata/pull/3553#discussion_r363569844
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java
 ##########
 @@ -983,8 +983,11 @@ public void print() {
     for (String levelSize : levels) {
       try {
         int size = Integer.parseInt(levelSize.trim());
-        if (validate(size, 100, 0, -1) < 0) {
+        if (validate(size, CarbonCommonConstants.COMPACTION_SEGMENT_LEVEL_MAX,
+            CarbonCommonConstants.COMPACTION_SEGMENT_LEVEL_MIN, -1) < 0) {
           // if given size is out of boundary then take default value for all levels.
+          LOGGER.warn("The COMPACTION_LEVEL_THRESHOLD is out of boundary, "
 
 Review comment:
   print the default value also, similar log like line : 995

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold

GitBox
In reply to this post by GitBox
ajantha-bhat commented on a change in pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold
URL: https://github.com/apache/carbondata/pull/3553#discussion_r363570106
 
 

 ##########
 File path: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/alterTable/TestAlterTableCompactionLevel.scala
 ##########
 @@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.spark.testsuite.alterTable
+
+import org.apache.spark.sql.CarbonEnv
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+/**
+ * test class for validating alter table set properties with alter_compaction_level properties
+ */
+class TestAlterTableCompactionLevel extends QueryTest with BeforeAndAfterAll {
+
+  private def isExpectedValueValid(dbName: String,
+                                   tableName: String,
+                                   key: String,
+                                   expectedValue: String): Boolean = {
+    val carbonTable = CarbonEnv.getCarbonTable(Option(dbName), tableName)(sqlContext.sparkSession)
+    val value = carbonTable.getTableInfo.getFactTable.getTableProperties.get(key)
+    expectedValue.equals(value)
+  }
+
+  private def dropTable = {
+    sql("drop table if exists alter_compaction_level")
+  }
+
+  override def beforeAll {
+    // drop table
+    dropTable
+    // create table
+    sql("create table alter_compaction_level(c1 String) stored by 'carbondata'")
+  }
+
+  test("validate alter compaction level") {
 
 Review comment:
   can you add a test case for more than 10000 threshold?
   
   As now there is no validation is present, it will pass !!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] marchpure commented on issue #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold

GitBox
In reply to this post by GitBox
marchpure commented on issue #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold
URL: https://github.com/apache/carbondata/pull/3553#issuecomment-577453860
 
 
   retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] marchpure commented on a change in pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold

GitBox
In reply to this post by GitBox
marchpure commented on a change in pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold
URL: https://github.com/apache/carbondata/pull/3553#discussion_r369882978
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
 ##########
 @@ -802,6 +802,17 @@ private CarbonCommonConstants() {
    */
   public static final String DEFAULT_SEGMENT_LEVEL_THRESHOLD = "4,3";
 
+
+  /**
+   * Min value of Number of segments to be merged in levels
 
 Review comment:
   solved

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] marchpure commented on a change in pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold

GitBox
In reply to this post by GitBox
marchpure commented on a change in pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold
URL: https://github.com/apache/carbondata/pull/3553#discussion_r369882999
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/util/AlterTableUtil.scala
 ##########
 @@ -533,6 +533,7 @@ object AlterTableUtil {
       "COMMENT",
       "COLUMN_META_CACHE",
       "CACHE_LEVEL",
+      "COMPACTION_LEVEL_THRESHOLD",
 
 Review comment:
   solved

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] marchpure commented on a change in pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold

GitBox
In reply to this post by GitBox
marchpure commented on a change in pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold
URL: https://github.com/apache/carbondata/pull/3553#discussion_r369883023
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java
 ##########
 @@ -983,8 +983,11 @@ public void print() {
     for (String levelSize : levels) {
       try {
         int size = Integer.parseInt(levelSize.trim());
-        if (validate(size, 100, 0, -1) < 0) {
+        if (validate(size, CarbonCommonConstants.COMPACTION_SEGMENT_LEVEL_MAX,
+            CarbonCommonConstants.COMPACTION_SEGMENT_LEVEL_MIN, -1) < 0) {
           // if given size is out of boundary then take default value for all levels.
+          LOGGER.warn("The COMPACTION_LEVEL_THRESHOLD is out of boundary, "
 
 Review comment:
   solved

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] marchpure commented on a change in pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold

GitBox
In reply to this post by GitBox
marchpure commented on a change in pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold
URL: https://github.com/apache/carbondata/pull/3553#discussion_r369883046
 
 

 ##########
 File path: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/alterTable/TestAlterTableCompactionLevel.scala
 ##########
 @@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.spark.testsuite.alterTable
+
+import org.apache.spark.sql.CarbonEnv
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+/**
+ * test class for validating alter table set properties with alter_compaction_level properties
+ */
+class TestAlterTableCompactionLevel extends QueryTest with BeforeAndAfterAll {
+
+  private def isExpectedValueValid(dbName: String,
+                                   tableName: String,
+                                   key: String,
+                                   expectedValue: String): Boolean = {
+    val carbonTable = CarbonEnv.getCarbonTable(Option(dbName), tableName)(sqlContext.sparkSession)
+    val value = carbonTable.getTableInfo.getFactTable.getTableProperties.get(key)
+    expectedValue.equals(value)
+  }
+
+  private def dropTable = {
+    sql("drop table if exists alter_compaction_level")
+  }
+
+  override def beforeAll {
+    // drop table
+    dropTable
+    // create table
+    sql("create table alter_compaction_level(c1 String) stored by 'carbondata'")
+  }
+
+  test("validate alter compaction level") {
 
 Review comment:
   solved

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA1 commented on issue #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold

GitBox
In reply to this post by GitBox
CarbonDataQA1 commented on issue #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold
URL: https://github.com/apache/carbondata/pull/3553#issuecomment-577471818
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1743/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] marchpure commented on issue #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold

GitBox
In reply to this post by GitBox
marchpure commented on issue #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold
URL: https://github.com/apache/carbondata/pull/3553#issuecomment-577474699
 
 
   retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA1 commented on issue #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold

GitBox
In reply to this post by GitBox
CarbonDataQA1 commented on issue #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold
URL: https://github.com/apache/carbondata/pull/3553#issuecomment-577490642
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1745/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] jackylk commented on issue #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold

GitBox
In reply to this post by GitBox
jackylk commented on issue #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold
URL: https://github.com/apache/carbondata/pull/3553#issuecomment-577588661
 
 
   LGTM

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] asfgit closed pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold

GitBox
In reply to this post by GitBox
asfgit closed pull request #3553: [CARBONDATA-3648] Support Alter Table Compaction Level Threshold
URL: https://github.com/apache/carbondata/pull/3553
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services
12