Re: [DISCUSSION] Forceful minor Compaction

Posted by Kunal Kapoor on
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/DISCUSSION-Forceful-minor-Compaction-tp10924p11320.html

Hi Liang,
Currently the user cannot choose which segments should be merged or not.
We should give a DDL which supports both forceful minor and major
compaction separately.

Another point would be if the user can specify the segments which he needs
merged. Obviously the segments that can be given should meet some criteria.
for example:-- no major compacted or 2nd Level compacted segments can be
provided during forced minor compaction.


--
Regards

*Kunal Kapoor*
Software Consultant

*Knoldus Software LLP <http://www.knoldus.com/home.knol>*
Twitter <http://twitter/kunal642> | LinkedIn
<https://in.linkedin.com/in/kunal642> | FB
<https://www.facebook.com/kunalkapoor642>
Mobile: [phone]

On Wed, Apr 19, 2017 at 2:33 PM, Liang Chen <[hidden email]> wrote:

> Hi Kunal
>
> Thank you for taking the good topic for discussion.
> First , let us think about : why users want to do forceful minor
> compaction,
> which cases?
> Current "MAJOR compaction" whether can cover "forceful MINOR compaction"
> scenarios ?
>
> As we know, compaction is mainly for optimizing index effectiveness by
> merging for multiple times data loading(segments), so current system
> provides two options for users to choose :"soft compaction(minor) and
> strong
> compaction(major)"。
>
> So i could not find the typical cases to add a new strong
> compaction(forceful minor)。
>
> Regards
> Liang
>
>
> Kunal Kapoor wrote
> > Hi all,
> > I was looking into compaction and had a query regarding the same.
> > If you have auto compaction turned on and the threshold level is 4,3
> > Now try loading data 7 times which will make 7 segments. Now as the auto
> > compaction was on therefore 4 segments will be merged to 0.1.
> > The segments visible will be 0.1, 4, 5, 6.
> >
> > When i try to run the compaction command nothing happens because the
> > threshold level is not reached for compaction to happen.
> > What if i want to merge the 3 segments i.e (4, 5, 6) and make a level-1
> > compacted segment.
> >
> > The proposed solution would be to add a new option to the compaction
> > command which contains what level of compaction the user would like to do
> >
> > Example:- alter table carbon_table compact 'minor' level '1'.
> > This would forcefully combine the segments(4, 5, 6) in to a level 1
> > compacted segment called 4.1 giving me 2 level-1 compacted segment (0.1
> > and
> > 4.1)
> > Similar operation can be done with level-1 compacted segments.
>
>
>
>
>
> --
> View this message in context: http://apache-carbondata-
> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
> Forceful-minor-Compaction-tp10924p11319.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive
> at Nabble.com.
>