[jira] [Updated] (CARBONDATA-3834) Segment directory and segment file in metadata are not created for partitioned table when 'carbon.merge.index.in.segment' property is set to false.

[Akash R Nilugal (Jira)]

     [ https://issues.apache.org/jira/browse/CARBONDATA-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Venugopal Reddy K updated CARBONDATA-3834:
------------------------------------------
    Description:
*[Issue]*

With the latest version of Carbon, Segment directory and segment file in metadata directory are not created for partitioned table when 'carbon.merge.index.in.segment' property is set to 'false'. And actual index files which were present in respective partition's '.tmp' directory are also deleted without moving them out to respective partition directory where its '.carbondata' file exist.

This issue was introduced from the resolution of an older optimization issue CARBONDATA-3641 +[Should improve data loading performance for partition table]+

i.e., with PR 3535

 

*[Modification Suggestion]*

 

If 'carbon.merge.index.in.segment' property is false, we can create the segment directory and segment file, and move the index file from respective partition's temp directory to partition directory where the .carbondata file exists.

Note: This need to be done before the respective partition's .tmp directory is deleted. Otherwise, we loose the index files.

  was:
*[Issue]*

With the latest version of Carbon, Segment directory and segment file in metadata directory are not created for partitioned table when 'carbon.merge.index.in.segment' property is set to 'false'. And actual index files which were present in respective partition's '.tmp' directory are also deleted without moving them out to respective partition directory where its '.carbondata' file exist.

This issue was introduced from the resolution of an older optimization issue CARBONDATA-3641 +[++Should improve data loading performance for partition table]+

 

i.e., PR 3535

*[Modification Suggestion]*

 

If 'carbon.merge.index.in.segment' property is false, we can create the segment directory and segment file, and move the index file from respective partition's temp directory to partition directory where the .carbondata file exists.

Note: This need to be done before the respective partition's .tmp directory is deleted. Otherwise, we loose the index files.

> Segment directory and segment file in metadata are not created for partitioned table when 'carbon.merge.index.in.segment' property is set to false.
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-3834
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-3834
>             Project: CarbonData
>          Issue Type: Bug
>          Components: hadoop-integration, spark-integration
>    Affects Versions: 2.0.0
>            Reporter: Venugopal Reddy K
>            Priority: Major
>
> *[Issue]*
> With the latest version of Carbon, Segment directory and segment file in metadata directory are not created for partitioned table when 'carbon.merge.index.in.segment' property is set to 'false'. And actual index files which were present in respective partition's '.tmp' directory are also deleted without moving them out to respective partition directory where its '.carbondata' file exist.
> This issue was introduced from the resolution of an older optimization issue CARBONDATA-3641 +[Should improve data loading performance for partition table]+
> i.e., with PR 3535
>  
> *[Modification Suggestion]*
>  
> If 'carbon.merge.index.in.segment' property is false, we can create the segment directory and segment file, and move the index file from respective partition's temp directory to partition directory where the .carbondata file exists.
> Note: This need to be done before the respective partition's .tmp directory is deleted. Otherwise, we loose the index files.

--
This message was sent by Atlassian Jira
(v8.3.4#803005)