Re: [Discussion]Do we still need to support carbon.merge.index.in.segment property ?

Posted by akashrn5 on Jul 09, 2020; 3:58pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Discussion-Do-we-still-need-to-support-carbon-merge-index-in-segment-property-tp97209p97225.html

Hi,

+1, we can deprecate it and as Vishal suggested we can keep as internal
property for developer purpose.

Regards,
Akash R Nilugal

On Thu, Jul 9, 2020, 2:46 PM VenuReddy <[hidden email]> wrote:

> Dear Community.!
>
> Have recently encountered a problem of Segment directory and segment file
> in
> metadata directory are not created for partitioned table when
> 'carbon.merge.index.in.segment' property is set to 'false'. And actual
> index
> files which were present in respective partition's '.tmp' directory are
> also
> deleted without moving them out to respective partition directory where its
> '.carbondata' file exist. Thus queries throw exception while reading index
> files. Please refer jira issue -
> https://issues.apache.org/jira/browse/CARBONDATA-3834
> <https://issues.apache.org/jira/browse/CARBONDATA-3834>
>
> To address this issue, we have 2 options to go with -
> 1. Either fix it to work for 'carbon.merge.index.in.segment' set to false
> case. There is an open PR  https://github.com/apache/carbondata/pull/3776
> <https://github.com/apache/carbondata/pull/3776>   for it.
>
> 2. Or Deprecate the 'carbon.merge.index.in.segment' property itself. As
> the
> query performance is better when merge index files are in use when compared
> to normal index files, and default behavior is to generate merge index
> files, probably it is not necessary to support
> 'carbon.merge.index.in.segment' anymore.
>
> What do you think about it ? Please let me know your opinion.
>
> Thanks,
> Venu
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>