Re: Clean files enhancement

Posted by BrooksLi on
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Clean-files-enhancement-tp100088p100453.html

Hi Vikram,

I agree to build a trash folder, +1.

Currently, the data loading/compaction/update/merge flow has automatic
cleaning files actions, but they are written separately.  Most of them are
aimed at deleting the stale segments(MARKED_FOR_DELETE/COMPACTED). And they
rely on the precise of the table status. If you could build a general clean
file function, it can be applied to substitute the current automatic
deletion for stale folders.

Besides, having a trash folder handle by Carbondata will be good, we can
find the deleted segments by this API.

And I think we should also consider the status of INSERT_IN_PROGERSS &
INSERT_OVERWRITE _IN_PROGRESS




--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/