[GitHub] [carbondata] Indhumathi27 opened a new pull request #4090: [WIP] Use CarbonFile API instead of java File API while writing Flink data

[GitBox]

Indhumathi27 opened a new pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090


    ### Why is this PR needed?
   
   
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-774671125


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5438/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-774671972


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3677/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-774700250


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5439/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-774701297


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3678/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-774971494


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5440/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-774975863


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3679/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#discussion_r572953288



##########
File path: docs/flink-integration-guide.md
##########
@@ -78,7 +78,7 @@ limitations under the License.
     val carbonProperties = new Properties
     // Set the carbon properties here, such as date format, store location, etc.
     
-    // Create carbon bulk writer factory. Two writer types are supported: 'Local' and 'S3'.
+    // Create carbon bulk writer factory. Three writer types are supported: 'Local', Hdfs' and 'S3'.

Review comment:
       Everywhere if we use file factory API and support HDFS conf input, only one writer is enough right ? do we need 3 writers ?
   Because in carbon table or SDK we don't create multiple type of writers to handle this kind of scenario.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Indhumathi27 commented on a change in pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#discussion_r572968625



##########
File path: docs/flink-integration-guide.md
##########
@@ -78,7 +78,7 @@ limitations under the License.
     val carbonProperties = new Properties
     // Set the carbon properties here, such as date format, store location, etc.
     
-    // Create carbon bulk writer factory. Two writer types are supported: 'Local' and 'S3'.
+    // Create carbon bulk writer factory. Three writer types are supported: 'Local', Hdfs' and 'S3'.

Review comment:
       yes. i also thought about the same. But since, already they have implemented writers for LOCAL and S3 type, i have implemented for HDFS. But i can see, there are some differences only for S3 writer, some extra configurations are needed and they are not creating directory while writing stage directories in S3. you can check  CarbonS3Writer.commit.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Indhumathi27 commented on a change in pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#discussion_r573488319



##########
File path: docs/flink-integration-guide.md
##########
@@ -78,7 +78,7 @@ limitations under the License.
     val carbonProperties = new Properties
     // Set the carbon properties here, such as date format, store location, etc.
     
-    // Create carbon bulk writer factory. Two writer types are supported: 'Local' and 'S3'.
+    // Create carbon bulk writer factory. Three writer types are supported: 'Local', Hdfs' and 'S3'.

Review comment:
       Chaned code to use  CarbonLocalWriter itself to handle Local and Hdfs FileSystems. Please review




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-776532399


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5450/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-776532783


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3689/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090#issuecomment-776719317


   LGTM


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

asfgit closed pull request #4090:
URL: https://github.com/apache/carbondata/pull/4090


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]