Posted by
Akash R Nilugal (Jira) on
Oct 29, 2020; 1:05pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/jira-Updated-CARBONDATA-4034-Improve-the-time-consuming-of-Horizontal-Compaction-for-update-tp103002.html
[
https://issues.apache.org/jira/browse/CARBONDATA-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jiayu Shen updated CARBONDATA-4034:
-----------------------------------
Description:
In the update flow, horizontal compaction will be significantly slower when updating with a lot of segments(or a lot of blocks). There is a case whose costing is as shown in the log.
{code:java}
2020-10-10 09:38:10,466 | INFO | [OperationManager-Background-Pool-28] | Horizontal Update Compaction operation started for [ods_oms.oms_wh_outbound_order]
2020-10-10 09:50:25,718 | INFO | [OperationManager-Background-Pool-28] | Horizontal Update Compaction operation completed for [ods_oms.oms_wh_outbound_order].
2020-10-10 10:15:44,302 | INFO | [OperationManager-Background-Pool-28] | Horizontal Delete Compaction operation started for [ods_oms.oms_wh_outbound_order]
2020-10-10 10:15:54,874 | INFO | [OperationManager-Background-Pool-28] | Horizontal Delete Compaction operation completed for [ods_oms.oms_wh_outbound_order].{code}
In this PR, we optimize the process between second and third row of the log, by optimizing the method _performDeleteDeltaCompaction_ in horizontal compaction flow.
was:
In the update flow, horizontal compaction will be significantly slower when updating with a lot of segments(or a lot of blocks). There is a case whose costing is as shown in the log.
2020-10-10 09:38:10,466 | INFO | [OperationManager-Background-Pool-28] | Horizontal Update Compaction operation started for [ods_oms.oms_wh_outbound_order]
2020-10-10 09:50:25,718 | INFO | [OperationManager-Background-Pool-28] | Horizontal Update Compaction operation completed for [ods_oms.oms_wh_outbound_order].
2020-10-10 10:15:44,302 | INFO | [OperationManager-Background-Pool-28] | Horizontal Delete Compaction operation started for [ods_oms.oms_wh_outbound_order]
2020-10-10 10:15:54,874 | INFO | [OperationManager-Background-Pool-28] | Horizontal Delete Compaction operation completed for [ods_oms.oms_wh_outbound_order].
In this PR, we optimize the process between second and third row of the log, by optimizing the method _performDeleteDeltaCompaction_ in horizontal compaction flow.
> Improve the time-consuming of Horizontal Compaction for update
> --------------------------------------------------------------
>
> Key: CARBONDATA-4034
> URL:
https://issues.apache.org/jira/browse/CARBONDATA-4034> Project: CarbonData
> Issue Type: Bug
> Reporter: Jiayu Shen
> Priority: Minor
> Time Spent: 17h 10m
> Remaining Estimate: 0h
>
> In the update flow, horizontal compaction will be significantly slower when updating with a lot of segments(or a lot of blocks). There is a case whose costing is as shown in the log.
> {code:java}
> 2020-10-10 09:38:10,466 | INFO | [OperationManager-Background-Pool-28] | Horizontal Update Compaction operation started for [ods_oms.oms_wh_outbound_order]
> 2020-10-10 09:50:25,718 | INFO | [OperationManager-Background-Pool-28] | Horizontal Update Compaction operation completed for [ods_oms.oms_wh_outbound_order].
> 2020-10-10 10:15:44,302 | INFO | [OperationManager-Background-Pool-28] | Horizontal Delete Compaction operation started for [ods_oms.oms_wh_outbound_order]
> 2020-10-10 10:15:54,874 | INFO | [OperationManager-Background-Pool-28] | Horizontal Delete Compaction operation completed for [ods_oms.oms_wh_outbound_order].{code}
> In this PR, we optimize the process between second and third row of the log, by optimizing the method _performDeleteDeltaCompaction_ in horizontal compaction flow.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)