[GitHub] [carbondata] nihal0107 opened a new pull request #3878: [WIP]fixed insert into select hive cmd

[GitBox]

nihal0107 opened a new pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878


    ### Why is this PR needed?
   
   
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-669112179


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1876/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-669112822


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3615/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-671090661


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3668/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-671090835


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1929/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r467935312



##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -213,6 +223,45 @@ private QueryModel getQueryModel(Configuration configuration, String path)
         .build();
   }
 
+  private static CarbonTable getCarbonTable(Configuration tableProperties)

Review comment:
       Please move this method to HiveCarbonUtil class.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r467937194



##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -213,6 +223,45 @@ private QueryModel getQueryModel(Configuration configuration, String path)
         .build();
   }
 
+  private static CarbonTable getCarbonTable(Configuration tableProperties)
+      throws InvalidConfigurationException, IOException, SQLException {
+    String[] tableUniqueName = tableProperties.get("name").split("\\.");
+    String databaseName = tableUniqueName[0];
+    String tableName = tableUniqueName[1];
+    String tablePath = tableProperties.get(hive_metastoreConstants.META_TABLE_LOCATION);
+    String columns = tableProperties.get(hive_metastoreConstants.META_TABLE_COLUMNS);
+    String sortColumns = tableProperties.get("sort_columns");
+    String columnTypes = tableProperties.get(hive_metastoreConstants.META_TABLE_COLUMN_TYPES);
+    String partitionColumns =
+        tableProperties.get(hive_metastoreConstants.META_TABLE_PARTITION_COLUMNS);
+    String partitionColumnTypes =
+        tableProperties.get(hive_metastoreConstants.META_TABLE_PARTITION_COLUMN_TYPES);
+    if (partitionColumns != null) {
+      columns = columns + "," + partitionColumns;
+      columnTypes = columnTypes + ":" + partitionColumnTypes;
+    }
+    String[] columnTypeArray = HiveCarbonUtil.splitSchemaStringToArray(columnTypes);
+
+    AbsoluteTableIdentifier absoluteTableIdentifier = AbsoluteTableIdentifier
+        .from(tableProperties.get(hive_metastoreConstants.META_TABLE_LOCATION),
+            getDatabaseName(tableProperties), getTableName(tableProperties));
+    String schemaPath =
+        CarbonTablePath.getSchemaFilePath(absoluteTableIdentifier.getTablePath(), tableProperties);
+
+    CarbonTable carbonTable;
+    String carbonDataFile = CarbonUtil.getFilePathExternalFilePath(schemaPath, tableProperties);
+    if (carbonDataFile == null) {

Review comment:
       This method is called only when carbondataFile does not exist. Refer: https://github.com/apache/carbondata/pull/3878/files#diff-e4da0735e7ef96dca3a7bfedb5a8039dR96
   
   Please remove this uncessary listing and directly use the CarbonTable.buildFromTableInfo() to build carbon table.
   
   No need for inferSchema code also




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r467937442



##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -202,7 +212,7 @@ protected void setFilterPredicates(Configuration configuration, CarbonTable carb
   }
 
   private QueryModel getQueryModel(Configuration configuration, String path)
-      throws IOException, InvalidConfigurationException {
+      throws IOException, InvalidConfigurationException, SQLException {

Review comment:
       Which method is throwing SQLException?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r467938413



##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/util/HiveCarbonUtil.java
##########
@@ -155,7 +155,7 @@ public static CarbonLoadModel getCarbonLoadModel(String tableName, String databa
     return loadModel;
   }
 
-  private static TableInfo getTableInfo(String tableName, String databaseName, String location,
+  public static TableInfo getTableInfo(String tableName, String databaseName, String location,

Review comment:
       revert this change after moving getCarbonTable to HiveCarbonUtil




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

nihal0107 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r468359209



##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -202,7 +212,7 @@ protected void setFilterPredicates(Configuration configuration, CarbonTable carb
   }
 
   private QueryModel getQueryModel(Configuration configuration, String path)
-      throws IOException, InvalidConfigurationException {
+      throws IOException, InvalidConfigurationException, SQLException {

Review comment:
       getTableInfo() is throwing SQLException which is called from getCarbonTable().




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

nihal0107 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r468359293



##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/util/HiveCarbonUtil.java
##########
@@ -155,7 +155,7 @@ public static CarbonLoadModel getCarbonLoadModel(String tableName, String databa
     return loadModel;
   }
 
-  private static TableInfo getTableInfo(String tableName, String databaseName, String location,
+  public static TableInfo getTableInfo(String tableName, String databaseName, String location,

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

nihal0107 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r468359349



##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -213,6 +223,45 @@ private QueryModel getQueryModel(Configuration configuration, String path)
         .build();
   }
 
+  private static CarbonTable getCarbonTable(Configuration tableProperties)
+      throws InvalidConfigurationException, IOException, SQLException {
+    String[] tableUniqueName = tableProperties.get("name").split("\\.");
+    String databaseName = tableUniqueName[0];
+    String tableName = tableUniqueName[1];
+    String tablePath = tableProperties.get(hive_metastoreConstants.META_TABLE_LOCATION);
+    String columns = tableProperties.get(hive_metastoreConstants.META_TABLE_COLUMNS);
+    String sortColumns = tableProperties.get("sort_columns");
+    String columnTypes = tableProperties.get(hive_metastoreConstants.META_TABLE_COLUMN_TYPES);
+    String partitionColumns =
+        tableProperties.get(hive_metastoreConstants.META_TABLE_PARTITION_COLUMNS);
+    String partitionColumnTypes =
+        tableProperties.get(hive_metastoreConstants.META_TABLE_PARTITION_COLUMN_TYPES);
+    if (partitionColumns != null) {
+      columns = columns + "," + partitionColumns;
+      columnTypes = columnTypes + ":" + partitionColumnTypes;
+    }
+    String[] columnTypeArray = HiveCarbonUtil.splitSchemaStringToArray(columnTypes);
+
+    AbsoluteTableIdentifier absoluteTableIdentifier = AbsoluteTableIdentifier
+        .from(tableProperties.get(hive_metastoreConstants.META_TABLE_LOCATION),
+            getDatabaseName(tableProperties), getTableName(tableProperties));
+    String schemaPath =
+        CarbonTablePath.getSchemaFilePath(absoluteTableIdentifier.getTablePath(), tableProperties);
+
+    CarbonTable carbonTable;
+    String carbonDataFile = CarbonUtil.getFilePathExternalFilePath(schemaPath, tableProperties);
+    if (carbonDataFile == null) {

Review comment:
       done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

nihal0107 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r468359455



##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -213,6 +223,45 @@ private QueryModel getQueryModel(Configuration configuration, String path)
         .build();
   }
 
+  private static CarbonTable getCarbonTable(Configuration tableProperties)

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-671826659


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3693/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-671828688


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1954/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-673935828


   LGTM


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

asfgit closed pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]