[GitHub] [carbondata] nihal0107 opened a new pull request #3975: [CARBONDATA-3964] Added test case for select query without filter

[GitBox]

nihal0107 opened a new pull request #3975:
URL: https://github.com/apache/carbondata/pull/3975


    ### Why is this PR needed?
    Added test case for select or select count query without filter to not prune with multi thread.
   
    ### What changes were proposed in this PR?
    Added test case for select or select count query without filter to not prune with multi thread.
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - Yes
   
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3975:
URL: https://github.com/apache/carbondata/pull/3975#issuecomment-706161324


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4348/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3975:
URL: https://github.com/apache/carbondata/pull/3975#issuecomment-706162827


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2598/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3975:
URL: https://github.com/apache/carbondata/pull/3975#issuecomment-706161324






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Indhumathi27 commented on a change in pull request #3975:
URL: https://github.com/apache/carbondata/pull/3975#discussion_r503699146



##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/filterexpr/CountStarTestCase.scala
##########
@@ -65,6 +65,51 @@ class CountStarTestCase extends QueryTest with BeforeAndAfterAll {
         CarbonCommonConstants.ENABLE_QUERY_STATISTICS_DEFAULT)
   }
 
+  test("select query without filter should not be pruned with multi thread") {
+    val numOfThreadsForPruning = CarbonProperties.getNumOfThreadsForPruning
+    val carbonDriverPruningMultiThreadEnableFilesCount =
+      CarbonProperties.getDriverPruningMultiThreadEnableFilesCount
+    CarbonProperties.getInstance()
+      .addProperty(CarbonCommonConstants.CARBON_MAX_DRIVER_THREADS_FOR_BLOCK_PRUNING, "2")
+    CarbonProperties.getInstance()
+      .addProperty(CarbonCommonConstants.CARBON_DRIVER_PRUNING_MULTI_THREAD_ENABLE_FILES_COUNT, "1")
+    try {
+      sql("CREATE TABLE filtertestTables (ID int, date Timestamp, country String, " +
+        "name String, phonetype String, serialname String, salary int) " +
+        "STORED AS carbondata"
+      )
+      val csvFilePath = s"$resourcesPath/datanullmeasurecol.csv"
+      sql(
+        s"LOAD DATA LOCAL INPATH '" + csvFilePath + "' INTO TABLE " +

Review comment:
       can format these lines. reduce to one line




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

nihal0107 commented on a change in pull request #3975:
URL: https://github.com/apache/carbondata/pull/3975#discussion_r503703995



##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/filterexpr/CountStarTestCase.scala
##########
@@ -65,6 +65,51 @@ class CountStarTestCase extends QueryTest with BeforeAndAfterAll {
         CarbonCommonConstants.ENABLE_QUERY_STATISTICS_DEFAULT)
   }
 
+  test("select query without filter should not be pruned with multi thread") {
+    val numOfThreadsForPruning = CarbonProperties.getNumOfThreadsForPruning
+    val carbonDriverPruningMultiThreadEnableFilesCount =
+      CarbonProperties.getDriverPruningMultiThreadEnableFilesCount
+    CarbonProperties.getInstance()
+      .addProperty(CarbonCommonConstants.CARBON_MAX_DRIVER_THREADS_FOR_BLOCK_PRUNING, "2")
+    CarbonProperties.getInstance()
+      .addProperty(CarbonCommonConstants.CARBON_DRIVER_PRUNING_MULTI_THREAD_ENABLE_FILES_COUNT, "1")
+    try {
+      sql("CREATE TABLE filtertestTables (ID int, date Timestamp, country String, " +
+        "name String, phonetype String, serialname String, salary int) " +
+        "STORED AS carbondata"
+      )
+      val csvFilePath = s"$resourcesPath/datanullmeasurecol.csv"
+      sql(
+        s"LOAD DATA LOCAL INPATH '" + csvFilePath + "' INTO TABLE " +

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Indhumathi27 commented on a change in pull request #3975:
URL: https://github.com/apache/carbondata/pull/3975#discussion_r503706862



##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/filterexpr/CountStarTestCase.scala
##########
@@ -65,6 +65,47 @@ class CountStarTestCase extends QueryTest with BeforeAndAfterAll {
         CarbonCommonConstants.ENABLE_QUERY_STATISTICS_DEFAULT)
   }
 
+  test("select query without filter should not be pruned with multi thread") {
+    val numOfThreadsForPruning = CarbonProperties.getNumOfThreadsForPruning
+    val carbonDriverPruningMultiThreadEnableFilesCount =
+      CarbonProperties.getDriverPruningMultiThreadEnableFilesCount
+    CarbonProperties.getInstance()
+      .addProperty(CarbonCommonConstants.CARBON_MAX_DRIVER_THREADS_FOR_BLOCK_PRUNING, "2")
+    CarbonProperties.getInstance()
+      .addProperty(CarbonCommonConstants.CARBON_DRIVER_PRUNING_MULTI_THREAD_ENABLE_FILES_COUNT, "1")
+    try {
+      sql("CREATE TABLE filtertestTables (ID int, date Timestamp, country String, " +
+        "name String, phonetype String, serialname String, salary int) " +
+        "STORED AS carbondata"
+      )
+      val csvFilePath = s"$resourcesPath/datanullmeasurecol.csv"
+      sql(
+        s"LOAD DATA LOCAL INPATH '" + csvFilePath + "' INTO TABLE " +
+          s"filtertestTables OPTIONS('DELIMITER'= ',', 'FILEHEADER'= '')"
+      )
+      sql(
+        s"LOAD DATA LOCAL INPATH '" + csvFilePath + "' INTO TABLE " +
+          s"filtertestTables OPTIONS('DELIMITER'= ',', 'FILEHEADER'= '')"
+      )
+      checkAnswer(
+        sql("select ID, Country, name, phoneType, serialName from filtertestTables"),

Review comment:
       with filter only, it will go for multi thread flow. please check




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Indhumathi27 commented on a change in pull request #3975:
URL: https://github.com/apache/carbondata/pull/3975#discussion_r503706862



##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/filterexpr/CountStarTestCase.scala
##########
@@ -65,6 +65,47 @@ class CountStarTestCase extends QueryTest with BeforeAndAfterAll {
         CarbonCommonConstants.ENABLE_QUERY_STATISTICS_DEFAULT)
   }
 
+  test("select query without filter should not be pruned with multi thread") {
+    val numOfThreadsForPruning = CarbonProperties.getNumOfThreadsForPruning
+    val carbonDriverPruningMultiThreadEnableFilesCount =
+      CarbonProperties.getDriverPruningMultiThreadEnableFilesCount
+    CarbonProperties.getInstance()
+      .addProperty(CarbonCommonConstants.CARBON_MAX_DRIVER_THREADS_FOR_BLOCK_PRUNING, "2")
+    CarbonProperties.getInstance()
+      .addProperty(CarbonCommonConstants.CARBON_DRIVER_PRUNING_MULTI_THREAD_ENABLE_FILES_COUNT, "1")
+    try {
+      sql("CREATE TABLE filtertestTables (ID int, date Timestamp, country String, " +
+        "name String, phonetype String, serialname String, salary int) " +
+        "STORED AS carbondata"
+      )
+      val csvFilePath = s"$resourcesPath/datanullmeasurecol.csv"
+      sql(
+        s"LOAD DATA LOCAL INPATH '" + csvFilePath + "' INTO TABLE " +
+          s"filtertestTables OPTIONS('DELIMITER'= ',', 'FILEHEADER'= '')"
+      )
+      sql(
+        s"LOAD DATA LOCAL INPATH '" + csvFilePath + "' INTO TABLE " +
+          s"filtertestTables OPTIONS('DELIMITER'= ',', 'FILEHEADER'= '')"
+      )
+      checkAnswer(
+        sql("select ID, Country, name, phoneType, serialName from filtertestTables"),

Review comment:
       with filter only, it will go for multi thread flow. please check if it is going through it or not




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3975:
URL: https://github.com/apache/carbondata/pull/3975#issuecomment-707586350


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4395/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3975:
URL: https://github.com/apache/carbondata/pull/3975#issuecomment-707591364


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2644/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Indhumathi27 commented on pull request #3975:
URL: https://github.com/apache/carbondata/pull/3975#issuecomment-708175230


   LGTM


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

asfgit closed pull request #3975:
URL: https://github.com/apache/carbondata/pull/3975


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]