Hi all,
OBS is an Object-based Storage Service developed and maintained by HuaweiCloud. It provides large storage capacity and is capable of storing any type of file. OBS supports both S3-client and OBS-client to connect to the OBS server. Currently in CarbonData we are supporting HDFS and S3 as FileSystems and I am proposing to support OBSFS as another FileSystem because of a few drawbacks of S3FileSystem. CarbonData needs OBSFS instead of S3 because of two main reasons: 1. Append: While doing append first we have to read the whole object and then append to the object which is quite slow. 2. Atomic Rename: No atomic rename is there in S3 as also mentioned in Jira [CARBONDATA-2670]. So both these issues can be fixed if we use OBSFileSystem. Any suggestions from the community will be greatly appreciated. I would be uploading the design doc shortly. Thanks and regards Manish Nalla EI BigData Kernel, Huawei Technologies India Pvt. Ltd |
Hi Manish Nalla,
Thanks for proposing this feature. Please clarify me the below points. 1. What will be the grammar to store carbondata files to OBSFS? 2. As S3 does not support Concurrent Data Manipulation operations and file leasing mechanisms, will this be the same behavior for OBSFS? 3. What are the Authentication properties that has to be configured to store carbondata files on to OBSFS location? Hope all these points will be covered in design document. Regards, Indhumathi M -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ |
In reply to this post by ManishNalla1994
Hi Manish,
May i know which version of hadoop will provide OBSFS support. Is this file system already supported in recent hadoop releases? Thanks Sujith. On Tue, 19 Feb 2019 at 4:18 PM, manish nalla <[hidden email]> wrote: > Hi all, > > OBS is an Object-based Storage Service developed and maintained by > HuaweiCloud. It provides large storage capacity and is capable of storing > any type of file. OBS supports both S3-client and OBS-client to connect to > the OBS server. > > Currently in CarbonData we are supporting HDFS and S3 as FileSystems and I > am proposing to support OBSFS as another FileSystem because of a few > drawbacks of S3FileSystem. > > CarbonData needs OBSFS instead of S3 because of two main reasons: > 1. Append: While doing append first we have to read the whole object and > then append to the object which is quite slow. > 2. Atomic Rename: No atomic rename is there in S3 as also mentioned in Jira > [CARBONDATA-2670]. > > So both these issues can be fixed if we use OBSFileSystem. > > Any suggestions from the community will be greatly appreciated. I would be > uploading the design doc shortly. > > Thanks and regards > Manish Nalla > EI BigData Kernel, > Huawei Technologies India Pvt. Ltd > |
Free forum by Nabble | Edit this page |