[GitHub] [carbondata] ajantha-bhat opened a new pull request #3731: [WIP] presto carbon reader should use tablePath from hive catalog

[GitBox]

ajantha-bhat opened a new pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731


    ### Why is this PR needed?
    In upgrade scenarios of 1.6 to 2.0, when sparl.sql.warehouse is not configured.
   Hive storage location is not proper. so, presto carbon integration should use tablePath from hive storage instead of location.
   
    ### What changes were proposed in this PR?
   use tablePath instead of location from hive metatstroe table.
       
    ### Does this PR introduce any user interface change?
    - No
   
    ### Is any new testcase added?
    - No
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#issuecomment-621322822


   @QiangCai , @kunal642 : please check.
   handled for both prestodb and prestosql


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#issuecomment-621381896


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2893/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#issuecomment-621385076


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1177/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#issuecomment-621585703


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1178/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#issuecomment-621586745


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2894/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

jackylk commented on a change in pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#discussion_r418481378



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -109,8 +109,14 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
     if (!table.getStorage().getStorageFormat().getInputFormat().contains("carbon")) {
       return super.getSplits(transactionHandle, session, layoutHandle, splitSchedulingStrategy);
     }
-    String location = table.getStorage().getLocation();
-
+    String location;

Review comment:
       assign with table.getStorage().getSerdeParameters().get("tablePath") first and then update it if null




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

jackylk commented on a change in pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#discussion_r418481410



##########
File path: integration/presto/src/main/prestosql/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -118,8 +118,14 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
     if (!table.getStorage().getStorageFormat().getInputFormat().contains("carbon")) {
       return super.getSplits(transactionHandle, session, tableHandle, splitSchedulingStrategy);
     }
-    String location = table.getStorage().getLocation();
-
+    String location;

Review comment:
       assign with table.getStorage().getSerdeParameters().get("tablePath") first and then update it if null




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#discussion_r418484147



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -109,8 +109,14 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
     if (!table.getStorage().getStorageFormat().getInputFormat().contains("carbon")) {
       return super.getSplits(transactionHandle, session, layoutHandle, splitSchedulingStrategy);
     }
-    String location = table.getStorage().getLocation();
-
+    String location;

Review comment:
       done

##########
File path: integration/presto/src/main/prestosql/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -118,8 +118,14 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
     if (!table.getStorage().getStorageFormat().getInputFormat().contains("carbon")) {
       return super.getSplits(transactionHandle, session, tableHandle, splitSchedulingStrategy);
     }
-    String location = table.getStorage().getLocation();
-
+    String location;

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#issuecomment-622363414


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1191/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#issuecomment-622364934


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2908/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#issuecomment-622655678


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#issuecomment-622668026


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2909/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#issuecomment-622668055


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1192/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#issuecomment-624151716


   @kunal642 , @jackylk : please check


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ravipesala commented on a change in pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#discussion_r422159455



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -109,8 +109,15 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
     if (!table.getStorage().getStorageFormat().getInputFormat().contains("carbon")) {
       return super.getSplits(transactionHandle, session, layoutHandle, splitSchedulingStrategy);
     }
-    String location = table.getStorage().getLocation();
-
+    String location;
+    String tablePath = table.getStorage().getSerdeParameters().get("tablePath");

Review comment:
       it is better like
   ```
   String location = table.getStorage().getSerdeParameters().get("tablePath");
   if (StringUtils.isEmpty(location))  {
      location = table.getStorage().getLocation();
   }
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#discussion_r422165473



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -109,8 +109,15 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
     if (!table.getStorage().getStorageFormat().getInputFormat().contains("carbon")) {
       return super.getSplits(transactionHandle, session, layoutHandle, splitSchedulingStrategy);
     }
-    String location = table.getStorage().getLocation();
-
+    String location;
+    String tablePath = table.getStorage().getSerdeParameters().get("tablePath");

Review comment:
       I think you mean StringUtils.isEmpty(`tablePath`) not `location`.
   But yeah, got your intention. I will change




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#discussion_r422165473



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -109,8 +109,15 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
     if (!table.getStorage().getStorageFormat().getInputFormat().contains("carbon")) {
       return super.getSplits(transactionHandle, session, layoutHandle, splitSchedulingStrategy);
     }
-    String location = table.getStorage().getLocation();
-
+    String location;
+    String tablePath = table.getStorage().getSerdeParameters().get("tablePath");

Review comment:
       I think you mean StringUtils.isEmpty(`tablePath`) not `location`.
   But yeah, got your intention. I will change




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#discussion_r422166664



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -109,8 +109,15 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
     if (!table.getStorage().getStorageFormat().getInputFormat().contains("carbon")) {
       return super.getSplits(transactionHandle, session, layoutHandle, splitSchedulingStrategy);
     }
-    String location = table.getStorage().getLocation();
-
+    String location;
+    String tablePath = table.getStorage().getSerdeParameters().get("tablePath");

Review comment:
       got it. ok.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3731:
URL: https://github.com/apache/carbondata/pull/3731#discussion_r422170438



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -109,8 +109,15 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
     if (!table.getStorage().getStorageFormat().getInputFormat().contains("carbon")) {
       return super.getSplits(transactionHandle, session, layoutHandle, splitSchedulingStrategy);
     }
-    String location = table.getStorage().getLocation();
-
+    String location;
+    String tablePath = table.getStorage().getSerdeParameters().get("tablePath");

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]