Re: CarbonData File Deletion Hotfix

Posted by BrooksLi on Sep 16, 2020; 2:55am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/CarbonData-File-Deletion-Hotfix-tp100384p100452.html

Hello March,

I agree to take a hotfix for data deletion in loading and compaction flow,
+1.  

Deleting the INSERT_IN_PROGERSS and INSERT_OVERWRITE_IN_PROGRESS is a
dangerous activity, so these two kinds of segments should not be
automatically deleted.

As for MARKED_FOR_DELETE and COMPACTED status segments, these are stale
segments, but we can keep them in the file system until the user/admin calls
clean file action manually.  Since the deletion requires the precision of
the table status.

So my opinion is to remove all the automatic clean steps in
loading/compaction flow first to protect the data from being deleted
accidentally.



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/