[
https://issues.apache.org/jira/browse/CARBONDATA-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Venkata Ramana G updated CARBONDATA-2745:
-----------------------------------------
Description:
AtomicFileOperationImpl creates a temporary file and then renames the file to actual file name. This is risky in S3 storage as the file has to be deleted and then recreated.
S3 supports atomic file overwrite. hdfs rename is atomic, while overwrite is not atomic and can result in empty file read temporarily.
So separate implementations for both hdfs and S3 to ensure consistancy of overwrite and read
was:AtomicFileOperationImpl creates a temporary file and then renames the file to actual file name. This is risky in S3 storage as the file has to be deleted and then recreated.
> Add a seperate Impl for AtomicFileOperations for s3
> ---------------------------------------------------
>
> Key: CARBONDATA-2745
> URL:
https://issues.apache.org/jira/browse/CARBONDATA-2745> Project: CarbonData
> Issue Type: Sub-task
> Reporter: Kunal Kapoor
> Assignee: Kunal Kapoor
> Priority: Major
> Fix For: 1.4.1
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> AtomicFileOperationImpl creates a temporary file and then renames the file to actual file name. This is risky in S3 storage as the file has to be deleted and then recreated.
>
> S3 supports atomic file overwrite. hdfs rename is atomic, while overwrite is not atomic and can result in empty file read temporarily.
> So separate implementations for both hdfs and S3 to ensure consistancy of overwrite and read
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Locked
