[jira] [Created] (CARBONDATA-3834) Segment directory and segment file in metadata are not created for partitioned table when 'carbon.merge.index.in.segment' property is set to false.

[Akash R Nilugal (Jira)]

Venugopal Reddy K created CARBONDATA-3834:
---------------------------------------------

             Summary: Segment directory and segment file in metadata are not created for partitioned table when 'carbon.merge.index.in.segment' property is set to false.
                 Key: CARBONDATA-3834
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-3834
             Project: CarbonData
          Issue Type: Improvement
          Components: hadoop-integration, spark-integration
    Affects Versions: 2.0.0
            Reporter: Venugopal Reddy K


*[Issue]*

With the latest version of Carbon, Segment directory and segment file in metadata directory are not created for partitioned table when 'carbon.merge.index.in.segment' property is set to 'false'. And actual index files which were present in respective partition's '.tmp' directory are also deleted without moving them out to respective partition directory where its '.carbondata' file exist.

This issue was introduced from the resolution of an older optimization issue CARBONDATA-3641 +[++Should improve data loading performance for partition table]+

 

i.e., PR 3535

*[Modification Suggestion]*

 

If 'carbon.merge.index.in.segment' property is false, we can create the segment directory and segment file, and move the index file from respective partition's temp directory to partition directory where the .carbondata file exists.

Note: This need to be done before the respective partition's .tmp directory is deleted. Otherwise, we loose the index files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)