Re: [carbondata-presto enhancements] support reading carbon SDK writer output in presto

Posted by Jacky Li on Dec 10, 2018; 3:12pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/carbondata-presto-enhancements-support-reading-carbon-SDK-writer-output-in-presto-tp69978p70106.html

Hi Ajantha,

Currently for carbon-presto integration, there is a plugin called “carbondata”. I wonder will you introduce new plugin into the project?
I suggest we re-use the same plugin and decide the read path within the plugin.
What do you think?

Regards,
Jacky

> 在 2018年12月10日，下午2:31，Ajantha Bhat <[hidden email]> 写道：
>
> Currently, carbon SDK files output (files without metadata folder and its
> contents) are read by spark using an external table with carbon session.
> But presto carbon integration doesn't support that. It can currently read
> only the transactional table output files.
>
> Hence we can enhance presto to read SDK output files. This will increase
> the use cases for presto-carbon integration.
>
> The above scenario can be achieved by inferring schema if metadata folder
> not exists and
> setting read committed scope to LatestFilesReadCommittedScope, if
> non-transctional table output files are present.
>
>
> Thanks,
> Ajantha
>