[DISCUSSION] Support Standard Spark's FileFormat interface in Carbondata

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSSION] Support Standard Spark's FileFormat interface in Carbondata

ravipesala
Hi,

Current Carbondata has deep integration with Spark to provide optimizations
in performance and also supports features like compaction, IUD, data maps
and metadata management etc. This type of integration forces user to use
CarbonSession instance to use carbon even for read and write operations.

So I am proposing standard spark's FileFormat implementation in carbon for
simple integration with Spark. Please check the jira for the design
document.
https://issues.apache.org/jira/browse/CARBONDATA-2872

--
Thanks & Regards,
Ravindra
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support Standard Spark's FileFormat interface in Carbondata

Liang Chen
Administrator
HI

+1, agree to support standard spark file format interface in carbondata, it
will be significantly helpful for broadening apache carbondata's ecosystem.

Regards
Liang


ravipesala wrote

> Hi,
>
> Current Carbondata has deep integration with Spark to provide
> optimizations
> in performance and also supports features like compaction, IUD, data maps
> and metadata management etc. This type of integration forces user to use
> CarbonSession instance to use carbon even for read and write operations.
>
> So I am proposing standard spark's FileFormat implementation in carbon for
> simple integration with Spark. Please check the jira for the design
> document.
> https://issues.apache.org/jira/browse/CARBONDATA-2872
>
> --
> Thanks & Regards,
> Ravindra





--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support Standard Spark's FileFormat interface in Carbondata

kumarvishal09
+1
-Regards
Kumar Vishal

On Thu, Aug 23, 2018 at 9:43 PM Liang Chen <[hidden email]> wrote:

> HI
>
> +1, agree to support standard spark file format interface in carbondata, it
> will be significantly helpful for broadening apache carbondata's ecosystem.
>
> Regards
> Liang
>
>
> ravipesala wrote
> > Hi,
> >
> > Current Carbondata has deep integration with Spark to provide
> > optimizations
> > in performance and also supports features like compaction, IUD, data maps
> > and metadata management etc. This type of integration forces user to use
> > CarbonSession instance to use carbon even for read and write operations.
> >
> > So I am proposing standard spark's FileFormat implementation in carbon
> for
> > simple integration with Spark. Please check the jira for the design
> > document.
> > https://issues.apache.org/jira/browse/CARBONDATA-2872
> >
> > --
> > Thanks & Regards,
> > Ravindra
>
>
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>
kumar vishal
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support Standard Spark's FileFormat interface in Carbondata

manishgupta88
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support Standard Spark's FileFormat interface in Carbondata

xm_zzc
In reply to this post by ravipesala
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support Standard Spark's FileFormat interface in Carbondata

David CaiQiang
In reply to this post by ravipesala
+1



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Best Regards
David Cai
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support Standard Spark's FileFormat interface in Carbondata

kanaka
In reply to this post by ravipesala
+ 1  for Carbon FileFormat.


Thanks for the design details.  


Regards,
Kanaka




--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
-
Kanaka
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support Standard Spark's FileFormat interface in Carbondata

aaron
In reply to this post by Liang Chen
Does this means that we could call carbon in pyspark?




--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/