Implementing Streaming Ingestion Support in CarbonData

[prabhatkashyap]

Hello, everyone

We are going to start implementing streaming ingestion support in carbondata. We'll be using Scala and be following document available on dev community by Aniket Adnaik

http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-New-Feature-Streaming-Ingestion-into-CarbonData-td9724.html

Your suggestions are welcome here.

Regards,
Prabhat Kashyap

[Aniket Adnaik]

 HI Prabhat,

Sorry, I missed your email dated on May 15th.
Regarding streaming feature, we already have some portion of implementation
in place and some other things are being worked on. There is lot of other
Streaming Ingestion functionality that still needs to be implemented.
I have created a Jira 1072 (
https://issues.apache.org/jira/browse/CARBONDATA-1072)  with work breakdown
list.
Your contributions are welcome. Lets sync up on work items so that there is
no conflict of work.

Best Regards,
Aniket

On 15 May 2017 10:46 pm, "prabhatkashyap" <[hidden email]>
wrote:

> Hello, everyone
>
> We are going to start implementing streaming ingestion support in
> carbondata. We'll be using Scala and be following document available on dev
> community by Aniket Adnaik
>
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5
> .nabble.com/DISCUSSION-New-Feature-Streaming-Ingestion-into-
> CarbonData-td9724.html
>
> Your suggestions are welcome here.
>
> Regards,
> Prabhat Kashyap
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-m
> ailing-list-archive.1130556.n5.nabble.com/Implementing-Strea
> ming-Ingestion-Support-in-CarbonData-tp12709.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>

[prabhatkashyap]

Hi Aniket,

We have started with Phase 1's first task that is Write Path Implementation. Please suggest if it is ok for us to continue on this. If it already in progress then please suggest what tasks can be own.
For better sync, I would suggest creating a branch for streaming where we can commit our code for streaming.

We would like to contribute on streaming functionality in Carbondata.

Regards,
Prabhat Kashyap

[Aniket Adnaik]

Hi Prabhat,

Phase-1 : *write path implementation is close to getting done with couple
of pending things that will leverage load path improvements in new release.
I may need some help in data conversion part.
I would say you guys can start focusing on work item-2 onwards (read path,
compaction, metadata management etc.).

Meanwhile, let me see if I can create a separate branch for streaming
ingest feature phase-1 with my changes. The current changes I have are
based off of old incubator branch in my local branch, will need to do
additional merge. I'll let you know soon.

I guess we work in different time zones (US, India), may need little extra
communication efforts to sync up :).

Best Regards,
Aniket








On Mon, May 22, 2017 at 12:54 AM, prabhatkashyap <[hidden email]
> wrote:

> Hi Aniket,
>
> We have started with Phase 1's first task that is *Write Path
> Implementation*. Please suggest if it is ok for us to continue on this. If
> it already in progress then please suggest what tasks can be own.
> For better sync, I would suggest creating a branch for streaming where we
> can commit our code for streaming.
>
> We would like to contribute on streaming functionality in Carbondata.
>
> Regards,
> Prabhat Kashyap
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/Implementing-
> Streaming-Ingestion-Support-in-CarbonData-tp12709p13061.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>

[Liang Chen-2]

Hi Aniket and Prabhat

Thanks for you guys started streaming ingestion feature.
I have created one branch with name "streaming_ingest".  Suggest creating
an independent module under integration/streaming.

Regards
Liang

2017-05-22 15:54 GMT+08:00 prabhatkashyap <[hidden email]>:

> Hi Aniket,
>
> We have started with Phase 1's first task that is *Write Path
> Implementation*. Please suggest if it is ok for us to continue on this. If
> it already in progress then please suggest what tasks can be own.
> For better sync, I would suggest creating a branch for streaming where we
> can commit our code for streaming.
>
> We would like to contribute on streaming functionality in Carbondata.
>
> Regards,
> Prabhat Kashyap
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/Implementing-
> Streaming-Ingestion-Support-in-CarbonData-tp12709p13061.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>