[GitHub] [carbondata] yutaoChina opened a new pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write

[GitBox]

yutaoChina opened a new pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892


    ### write carbon file less than 1M , configured minimum value (dfs.namenode.fs-limits.min-block-size);can't write hdfs file
   
   
   
   
    ### the default set block size is 1024 byte and compare with  file size;when file size big than 1kb but less than 1M ,can not write file to hdfs ;so set default block size to 2M  
   
       
    ### Does this PR introduce any user interface change?
    - No
   
    ### Is any new testcase added?
    - No
   
   
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-674365676


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-674649559






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-674699373


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3741/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-674700072


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2001/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-674706635


   @yutaoChina : Thanks for working on this.
   a) please handle the compilation error
   b) please create a jira issue and add it in the issue header


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#discussion_r471286842



##########
File path: integration/flink/src/main/java/org/apache/carbon/core/metadata/StageManager.java
##########
@@ -81,7 +81,7 @@ public static void writeStageInput(final String stageInputPath, final StageInput
   private static void writeSuccessFile(final String successFilePath) throws IOException {
     final DataOutputStream segmentStatusSuccessOutputStream =
         FileFactory.getDataOutputStream(successFilePath,
-            CarbonCommonConstants.BYTEBUFFER_SIZE, 1024);
+            CarbonCommonConstants.BYTEBUFFER_SIZE, 1024 * 1024 * 2);

Review comment:
       what if the file size is greater than 2 MB  ? why 2MB selected for this ?
   may be need to pass the actual file size ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

yutaoChina commented on a change in pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#discussion_r472681680



##########
File path: integration/flink/src/main/java/org/apache/carbon/core/metadata/StageManager.java
##########
@@ -81,7 +81,7 @@ public static void writeStageInput(final String stageInputPath, final StageInput
   private static void writeSuccessFile(final String successFilePath) throws IOException {
     final DataOutputStream segmentStatusSuccessOutputStream =
         FileFactory.getDataOutputStream(successFilePath,
-            CarbonCommonConstants.BYTEBUFFER_SIZE, 1024);
+            CarbonCommonConstants.BYTEBUFFER_SIZE, 1024 * 1024 * 2);

Review comment:
       i set it 2M beacuase hdfs (dfs.namenode.fs-limits.min-block-size) configured minimum value size is 1M and in CarbonUtil.java class
   `getMaxOfBlockAndFileSize(long blockSize, long fileSize) `method use `long maxSize = blockSize;
       if (fileSize > blockSize) {
         maxSize = fileSize;
       }`
   if default size or filesize less than 1M program will get error ;
   why 2M ? default is 1M so default * 2 bigger than it




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

yutaoChina commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-675874179


   >
   >
   > @yutaoChina : Thanks for working on this.
   > a) please handle the compilation error
   > b) please create a jira issue and add it in the issue header
   this is jira issues
   https://issues.apache.org/jira/browse/CARBONDATA-3925


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677162461


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677439466


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3805/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677445838


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2064/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

yutaoChina commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677487173


   >
   >
   > @yutaoChina : Thanks for working on this.
   > a) please handle the compilation error
   > b) please create a jira issue and add it in the issue header
   
   my jira id is yutaochina


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677539801


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2068/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677550766


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3809/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677668597


   @yutaoChina : you can run
   `mvn clean install -Pspark-2.3 -Pbuild-with-format -DskipTests` in your IDE to find all findbugs, checkstyle, scalastyle issues.
   Now also it has some check style issue


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677734623


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2080/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677736438


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3821/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-678023200


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2086/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-678025176


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3827/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]