[GitHub] [carbondata] nihal0107 opened a new pull request #4042: [CARBONDATA-4069] handled set streaming for SI table or table having SI.

[GitBox]

nihal0107 opened a new pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042


    ### Why is this PR needed?
    Currently `alter table set streaming = true` is allowed either for SI table or MT which has SI.
   
    ### What changes were proposed in this PR?
   Handled set streaming true for SI table and MT which has SI and throws the exception.
       
    ### Does this PR introduce any user interface change?
    - No
   
    ### Is any new testcase added?
    - Yes
   
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-738658347






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-739468864


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5048/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-739480243


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3290/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-739808873


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-739817193


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3322/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-739818120


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5103/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-739818588


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5083/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-739818955


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3348/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

brijoobopanna commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-739833586


   retest this please
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-739846605


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3329/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-739896899


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5090/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

brijoobopanna commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-739927931


   retest this please
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-739990776


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5096/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-739992275


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3335/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-741694403


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5125/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-741747065


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5126/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#issuecomment-741758732


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3364/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Indhumathi27 commented on a change in pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#discussion_r539851556



##########
File path: integration/spark/src/main/scala/org/apache/spark/util/AlterTableUtil.scala
##########
@@ -916,6 +930,19 @@ object AlterTableUtil {
         }
       }
     }
+    // should not be present in index tables
+    val indexTables = CarbonIndexUtil.getSecondaryIndexes(carbonTable)

Review comment:
       Instead of getting from index table, can directly get the index Column names from carbonTable.getIndexesMap.get(IndexType.SI.getIndexProviderName) -> CarbonCommonConstants.INDEX_COLUMNS

##########
File path: integration/spark/src/main/scala/org/apache/spark/util/AlterTableUtil.scala
##########
@@ -450,14 +451,27 @@ object AlterTableUtil {
       // validate long string columns
       val longStringColumns = lowerCasePropertiesMap.get("long_string_columns");
       if (longStringColumns.isDefined) {
-        validateLongStringColumns(longStringColumns.get, carbonTable)
+        validateLongStringColumns(longStringColumns.get, carbonTable, sparkSession)
         // update schema for long string columns
         updateSchemaForLongStringColumns(thriftTable, longStringColumns.get)
       } else if (propKeys.exists(_.equalsIgnoreCase("long_string_columns") && !set)) {
         if (tblPropertiesMap.exists(prop => prop._1.equalsIgnoreCase("long_string_columns"))) {
           updateSchemaForLongStringColumns(thriftTable, "")
         }
       }
+      // validate set for streaming table
+      val streamingOption = lowerCasePropertiesMap.get("streaming")
+      if (streamingOption.isDefined && set) {
+        if (carbonTable.isIndexTable) {
+          throw new UnsupportedOperationException("Set streaming table is " +
+            "not allowed for index table.")

Review comment:
       ```suggestion
               "not allowed on the index table.")
   ```

##########
File path: integration/spark/src/main/scala/org/apache/spark/util/AlterTableUtil.scala
##########
@@ -916,6 +930,19 @@ object AlterTableUtil {
         }
       }
     }
+    // should not be present in index tables
+    val indexTables = CarbonIndexUtil.getSecondaryIndexes(carbonTable)
+    if (!indexTables.isEmpty) {
+      indexTables.asScala.map(indexTable => {
+        CarbonEnv.getCarbonTable(Some(carbonTable.getDatabaseName), indexTable)(sparkSession)
+          .getCreateOrderColumn.asScala.map(column => {
+          if (longStringCols.contains(column.getColName)) {
+            throw new MalformedCarbonCommandException(s"col ${column.getColName} is " +

Review comment:
       Can change to, "Cannot Alter column 'b' to Long_string_column, as the column exists in a secondary index with name `indexTable`.  LONG_STRING_COLUMNS is not allowed on secondary index."




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

nihal0107 commented on a change in pull request #4042:
URL: https://github.com/apache/carbondata/pull/4042#discussion_r540031120



##########
File path: integration/spark/src/main/scala/org/apache/spark/util/AlterTableUtil.scala
##########
@@ -450,14 +451,27 @@ object AlterTableUtil {
       // validate long string columns
       val longStringColumns = lowerCasePropertiesMap.get("long_string_columns");
       if (longStringColumns.isDefined) {
-        validateLongStringColumns(longStringColumns.get, carbonTable)
+        validateLongStringColumns(longStringColumns.get, carbonTable, sparkSession)
         // update schema for long string columns
         updateSchemaForLongStringColumns(thriftTable, longStringColumns.get)
       } else if (propKeys.exists(_.equalsIgnoreCase("long_string_columns") && !set)) {
         if (tblPropertiesMap.exists(prop => prop._1.equalsIgnoreCase("long_string_columns"))) {
           updateSchemaForLongStringColumns(thriftTable, "")
         }
       }
+      // validate set for streaming table
+      val streamingOption = lowerCasePropertiesMap.get("streaming")
+      if (streamingOption.isDefined && set) {
+        if (carbonTable.isIndexTable) {
+          throw new UnsupportedOperationException("Set streaming table is " +
+            "not allowed for index table.")

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]