[DISCUSSION] Support OBSFS

[ManishNalla1994]

Hi all,

OBS is an Object-based Storage Service developed and maintained by
HuaweiCloud. It provides large storage capacity and is capable of storing
any type of file. OBS supports both S3-client and OBS-client to connect to
the OBS server.

Currently in CarbonData we are supporting HDFS and S3 as FileSystems and I
am proposing to support OBSFS as another FileSystem because of a few
drawbacks of S3FileSystem.

CarbonData needs OBSFS instead of S3 because of two main reasons:
1. Append: While doing append first we have to read the whole object and
then append to the object which is quite slow.
2. Atomic Rename: No atomic rename is there in S3 as also mentioned in Jira
[CARBONDATA-2670].

So both these issues can be fixed if we use OBSFileSystem.

Any suggestions from the community will be greatly appreciated. I would be
uploading the design doc shortly.

Thanks and regards
Manish Nalla
EI BigData Kernel,
Huawei Technologies India Pvt. Ltd

[Indhumathi]

Hi Manish Nalla,

Thanks for proposing this feature. Please clarify me the below points.

1. What will be the grammar to store carbondata files to OBSFS?
2. As S3 does not support Concurrent Data Manipulation operations and
    file leasing mechanisms, will this be the same behavior for OBSFS?
3. What are the Authentication properties that has to be configured to
    store carbondata files on to OBSFS location?

Hope all these points will be covered in design document.

Regards,
Indhumathi M




--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[sujith chacko]

Hi Manish,

 May i know which version of hadoop will provide OBSFS support. Is this
file system already supported in recent hadoop releases?

Thanks
Sujith.

On Tue, 19 Feb 2019 at 4:18 PM, manish nalla <[hidden email]>
wrote:

> Hi all,
>
> OBS is an Object-based Storage Service developed and maintained by
> HuaweiCloud. It provides large storage capacity and is capable of storing
> any type of file. OBS supports both S3-client and OBS-client to connect to
> the OBS server.
>
> Currently in CarbonData we are supporting HDFS and S3 as FileSystems and I
> am proposing to support OBSFS as another FileSystem because of a few
> drawbacks of S3FileSystem.
>
> CarbonData needs OBSFS instead of S3 because of two main reasons:
> 1. Append: While doing append first we have to read the whole object and
> then append to the object which is quite slow.
> 2. Atomic Rename: No atomic rename is there in S3 as also mentioned in Jira
> [CARBONDATA-2670].
>
> So both these issues can be fixed if we use OBSFileSystem.
>
> Any suggestions from the community will be greatly appreciated. I would be
> uploading the design doc shortly.
>
> Thanks and regards
> Manish Nalla
> EI BigData Kernel,
> Huawei Technologies India Pvt. Ltd
>