Carbondata is not showing stats

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Carbondata is not showing stats

Rana Faisal Munir
Hi,

I am running some experiments with Parquet and Carbondata. I observe that Carbondata is not recording statistics for Spark to display in Spark UI (history server). I run a projection queries both on Parquet (sqlContext.sql()) and Carbondata (carbonContext.sql()). Parquet is showing the complete details (total tasks and # of disk read in bytes) whereas Carbondata is not showing anything. I would like to have these statistics for carbondata. It helps to see how much Carbondata is reading from the disk which is very useful while comparing it with other formats.

I have also attached screenshot to make it easy to understand. I am using latest Carbondata version apache-carbondata-1.1.0.


Regards
Faisal
Reply | Threaded
Open this post in threaded view
|

Re: Carbondata is not showing stats

ravipesala
Hi,

There is an issue in recording statistics in carbon, and it is being
addressed in PR https://github.com/apache/carbondata/pull/918 . So you can
verify in master after it got merged

Regards,
Ravindra
On Mon, 22 May 2017 at 11:09 PM, Rana Faisal <[hidden email]> wrote:

> Hi,
>
> I am running some experiments with Parquet and Carbondata. I observe that
> Carbondata is not recording statistics for Spark to display in Spark UI
> (history server). I run a projection queries both on Parquet
> (sqlContext.sql()) and Carbondata (carbonContext.sql()). Parquet is showing
> the complete details (total tasks and # of disk read in bytes) whereas
> Carbondata is not showing anything. I would like to have these statistics
> for carbondata. It helps to see how much Carbondata is reading from the
> disk which is very useful while comparing it with other formats.
>
> I have also attached screenshot to make it easy to understand. I am using
> latest Carbondata version apache-carbondata-1.1.0.
> <https://github.com/apache/carbondata/releases/tag/apache-carbondata-1.1.0>
>
>
> Regards
> Faisal
>