[DISCUSSION] Forceful minor Compaction

[Kunal Kapoor]

Hi all,
I was looking into compaction and had a query regarding the same.
If you have auto compaction turned on and the threshold level is 4,3
Now try loading data 7 times which will make 7 segments. Now as the auto
compaction was on therefore 4 segments will be merged to 0.1.
The segments visible will be 0.1, 4, 5, 6.

When i try to run the compaction command nothing happens because the
threshold level is not reached for compaction to happen.
What if i want to merge the 3 segments i.e (4, 5, 6) and make a level-1
compacted segment.

The proposed solution would be to add a new option to the compaction
command which contains what level of compaction the user would like to do

Example:- alter table carbon_table compact 'minor' level '1'.
This would forcefully combine the segments(4, 5, 6) in to a level 1
compacted segment called 4.1 giving me 2 level-1 compacted segment (0.1 and
4.1)
Similar operation can be done with level-1 compacted segments.

[Liang Chen]

Hi Kunal

Thank you for taking the good topic for discussion.
First , let us think about : why users want to do forceful minor compaction, which cases?
Current "MAJOR compaction" whether can cover "forceful MINOR compaction" scenarios ?

As we know, compaction is mainly for optimizing index effectiveness by merging for multiple times data loading(segments), so current system provides two options for users to choose :"soft compaction(minor) and strong compaction(major)"。

So i could not find the typical cases to add a new strong compaction(forceful minor)。

Regards
Liang

Kunal Kapoor wrote

Hi all,
I was looking into compaction and had a query regarding the same.
If you have auto compaction turned on and the threshold level is 4,3
Now try loading data 7 times which will make 7 segments. Now as the auto
compaction was on therefore 4 segments will be merged to 0.1.
The segments visible will be 0.1, 4, 5, 6.

When i try to run the compaction command nothing happens because the
threshold level is not reached for compaction to happen.
What if i want to merge the 3 segments i.e (4, 5, 6) and make a level-1
compacted segment.

The proposed solution would be to add a new option to the compaction
command which contains what level of compaction the user would like to do

Example:- alter table carbon_table compact 'minor' level '1'.
This would forcefully combine the segments(4, 5, 6) in to a level 1
compacted segment called 4.1 giving me 2 level-1 compacted segment (0.1 and
4.1)
Similar operation can be done with level-1 compacted segments.

[Kunal Kapoor]

Hi Liang,
Currently the user cannot choose which segments should be merged or not.
We should give a DDL which supports both forceful minor and major
compaction separately.

Another point would be if the user can specify the segments which he needs
merged. Obviously the segments that can be given should meet some criteria.
for example:-- no major compacted or 2nd Level compacted segments can be
provided during forced minor compaction.


--
Regards

*Kunal Kapoor*
Software Consultant

*Knoldus Software LLP <http://www.knoldus.com/home.knol>*
Twitter <http://twitter/kunal642> | LinkedIn
<https://in.linkedin.com/in/kunal642> | FB
<https://www.facebook.com/kunalkapoor642>
Mobile: [phone]

On Wed, Apr 19, 2017 at 2:33 PM, Liang Chen <[hidden email]> wrote:

> Hi Kunal
>
> Thank you for taking the good topic for discussion.
> First , let us think about : why users want to do forceful minor
> compaction,
> which cases?
> Current "MAJOR compaction" whether can cover "forceful MINOR compaction"
> scenarios ?
>
> As we know, compaction is mainly for optimizing index effectiveness by
> merging for multiple times data loading(segments), so current system
> provides two options for users to choose :"soft compaction(minor) and
> strong
> compaction(major)"。
>
> So i could not find the typical cases to add a new strong
> compaction(forceful minor)。
>
> Regards
> Liang
>
>
> Kunal Kapoor wrote
> > Hi all,
> > I was looking into compaction and had a query regarding the same.
> > If you have auto compaction turned on and the threshold level is 4,3
> > Now try loading data 7 times which will make 7 segments. Now as the auto
> > compaction was on therefore 4 segments will be merged to 0.1.
> > The segments visible will be 0.1, 4, 5, 6.
> >
> > When i try to run the compaction command nothing happens because the
> > threshold level is not reached for compaction to happen.
> > What if i want to merge the 3 segments i.e (4, 5, 6) and make a level-1
> > compacted segment.
> >
> > The proposed solution would be to add a new option to the compaction
> > command which contains what level of compaction the user would like to do
> >
> > Example:- alter table carbon_table compact 'minor' level '1'.
> > This would forcefully combine the segments(4, 5, 6) in to a level 1
> > compacted segment called 4.1 giving me 2 level-1 compacted segment (0.1
> > and
> > 4.1)
> > Similar operation can be done with level-1 compacted segments.
>
>
>
>
>
> --
> View this message in context: http://apache-carbondata-
> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
> Forceful-minor-Compaction-tp10924p11319.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive
> at Nabble.com.
>

[Venkata Gollamudi]

Forceful compaction might be required when user really wants control and
wanted to force trigger compaction on required segments.
How ever need to identify cases and its behavior , so that it can co exist
with current compaction model, with out disturbing or imposing any
limitations.
I feel forceful major compaction specifying segments will be better option
than forceful minor.

Regards,
Venkata Ramana Gollamudi


On Apr 19, 2017 14:50, "Kunal Kapoor" <[hidden email]> wrote:

> Hi Liang,
> Currently the user cannot choose which segments should be merged or not.
> We should give a DDL which supports both forceful minor and major
> compaction separately.
>
> Another point would be if the user can specify the segments which he needs
> merged. Obviously the segments that can be given should meet some criteria.
> for example:-- no major compacted or 2nd Level compacted segments can be
> provided during forced minor compaction.
>
>
> --
> Regards
>
> *Kunal Kapoor*
> Software Consultant
>
> *Knoldus Software LLP <http://www.knoldus.com/home.knol>*
> Twitter <http://twitter/kunal642> | LinkedIn
> <https://in.linkedin.com/in/kunal642> | FB
> <https://www.facebook.com/kunalkapoor642>
> Mobile: [phone]
>
> On Wed, Apr 19, 2017 at 2:33 PM, Liang Chen <[hidden email]>
> wrote:
>
> > Hi Kunal
> >
> > Thank you for taking the good topic for discussion.
> > First , let us think about : why users want to do forceful minor
> > compaction,
> > which cases?
> > Current "MAJOR compaction" whether can cover "forceful MINOR compaction"
> > scenarios ?
> >
> > As we know, compaction is mainly for optimizing index effectiveness by
> > merging for multiple times data loading(segments), so current system
> > provides two options for users to choose :"soft compaction(minor) and
> > strong
> > compaction(major)"。
> >
> > So i could not find the typical cases to add a new strong
> > compaction(forceful minor)。
> >
> > Regards
> > Liang
> >
> >
> > Kunal Kapoor wrote
> > > Hi all,
> > > I was looking into compaction and had a query regarding the same.
> > > If you have auto compaction turned on and the threshold level is 4,3
> > > Now try loading data 7 times which will make 7 segments. Now as the
> auto
> > > compaction was on therefore 4 segments will be merged to 0.1.
> > > The segments visible will be 0.1, 4, 5, 6.
> > >
> > > When i try to run the compaction command nothing happens because the
> > > threshold level is not reached for compaction to happen.
> > > What if i want to merge the 3 segments i.e (4, 5, 6) and make a level-1
> > > compacted segment.
> > >
> > > The proposed solution would be to add a new option to the compaction
> > > command which contains what level of compaction the user would like to
> do
> > >
> > > Example:- alter table carbon_table compact 'minor' level '1'.
> > > This would forcefully combine the segments(4, 5, 6) in to a level 1
> > > compacted segment called 4.1 giving me 2 level-1 compacted segment (0.1
> > > and
> > > 4.1)
> > > Similar operation can be done with level-1 compacted segments.
> >
> >
> >
> >
> >
> > --
> > View this message in context: http://apache-carbondata-
> > mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
> > Forceful-minor-Compaction-tp10924p11319.html
> > Sent from the Apache CarbonData Mailing List archive mailing list archive
> > at Nabble.com.
> >
>