Refactored SegmentPropertiesFetcher to resolve pruning problem post the carbon schema restructure.

[mohdshahidkhan]

Hi Dev,

Currently DataMap while pruning is considering SegmentProperties from the
0th block of Segment DataMap.
It is not correct as post schema restructure if the table is updated
(*Update/Delete record*) then within the segment, all the block will not
have symmetric schema.

To support schema restructure, it must be ensured that the block is pruned
with its own columns schema.


Regards,
Shahid



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[xuchuanyin]

Will delete/update affect the schema?

What's the meaning of 'schema' here?



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[ravipesala]

Yes Shahid, you are right. During Update scenario there is a chance of
creating new data within the same segment. And it will lead to wrong data if
the schema is different. It is not same case for delete as we don't have any
schema for it.

I feel we always better create a new segment even for update if any new data
is added instead of updating the same segment.



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/