[GitHub] [carbondata] akashrn5 opened a new pull request #3733: [WIP]fix cache issue in case of compaction failure in compaction post listeners

[GitBox]

akashrn5 opened a new pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733


   
   
    ### Why is this PR needed?
   
   
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733#issuecomment-621745957


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2904/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733#issuecomment-621746890


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1187/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733#issuecomment-621820973


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1190/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733#issuecomment-621823501


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2907/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733#issuecomment-623390821


   @akashrn5 : Issue is happening when a segment is removed from main table but cache is not cleaned right?
   
   When the SI table compaction disable why not just disable SI, why need to remove whole segment of main table ?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

akashrn5 commented on pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733#issuecomment-623396522


   > @akashrn5 : Issue is happening when a segment is removed from main table but cache is not cleaned right?
   >
   > When the SI table compaction disable why not just disable SI, why need to remove whole segment of main table ?
   
   @ajantha-bhat , once the main table compaction is finished, say example, 0.1 for segments 1, 2, then 0.1 segment data of the main table is selected and loaded to SI table's 0.1 segment. So if while loading 0.1 to SI fails, then 0.1 entry won't be present in the main table, and the cache will have this entry for the main table.
   
   So when next time again the same 0.1 is compacted for 0,1 segments as it was failed before, in SI it will fire a select query to 0.1 of the main table, since the cache is not cleared, it will try for old carbondata files and say file not found.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733#issuecomment-623408141


   > So if while loading 0.1 to SI fails, then 0.1 entry won't be present in the main table
   
   This is what I mean, why we need to fail the main table compaction when SI compaction fails ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat edited a comment on pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733#issuecomment-623408141


   > So if while loading 0.1 to SI fails, then 0.1 entry won't be present in the main table
   
   This is what I mean, why we need to fail the main table compaction when SI compaction fails ? We can disable the SI


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat edited a comment on pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733#issuecomment-623408141


   > So if while loading 0.1 to SI fails, then 0.1 entry won't be present in the main table
   
   This is what I mean, why we need to fail the main table compaction when SI compaction fails ? We can disable the SI.
   
   we can load SI separately from compacted segments later from rebuild/refresh command


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

akashrn5 commented on pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733#issuecomment-623414736


   > we can load SI separately from compacted segments later from rebuild/refresh command
   
   we do not have any refresh command for SI FYI. as we haven't made it like the other indexed on carbon table. So this is the old behavior.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733#issuecomment-623415702


   compacting whole main table again will take huge time, as maintable already compaction is success we should handle like other datamaps.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

akashrn5 commented on pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733#issuecomment-623416505


   > compacting whole main table again will take huge time, as maintable already compaction is success we should handle like other datamaps.
   
   main table compaction is success functionally, but not yet committed as success segment in metadata. So handling like other datamaps is altogether new requirement. cannot handle here.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3733:
URL: https://github.com/apache/carbondata/pull/3733#issuecomment-624627760


   ok. LGTM


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]