http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/CarbonData-Performance-Optimization-tp62950p63956.html
> +1
>
> > 在 2018年9月21日，上午10:20，Ravindra Pesala <
[hidden email]> 写道：
> >
> > Hi,
> >
> > In case of querying data using Spark or Presto, carbondata is not well
> > optimized for reading data and fill the vector. The major issues are as
> > follows.
> > 1. CarbonData has long method stack for reading and filling out the data
> to
> > vector.
> > 2. Many conditions and checks before filling out the data to vector.
> > 3. Maintaining intermediate copies of data leads more CPU utilization.
> > Because of the above issues, there is a high chance of missing the CPU
> > cache while processing the leads to poor performance.
> >
> > So here I am proposing the optimization to fill the vector without much
> > method stack and condition checks and no intermediate copies to utilize
> > more CPU cache.
> >
> > *Full Scan queries:*
> > After decompressing the page in our V3 reader we can immediately fill
> the
> > data to a vector without any condition checks inside loops. So here
> > complete column page data is set to column vector in a single batch and
> > gives back data to Spark/Presto.
> > *Filter Queries:*
> > First, apply page level pruning using the min/max of each page and get
> > the valid pages of blocklet. Decompress only valid pages and fill the
> > vector directly as mentioned in full scan query scenario.
> >
> > In this method, we can also get the advantage of avoiding two times
> > filtering in Spark/Presto as they do the filtering again even though we
> > return the filtered data.
> >
> > Please find the *TPCH performance report of updated carbon* as per the
> > changes mentioned above. Please note that the changes I have done the
> > changes in POC quality so it takes some time to stabilize it.
> >
> > *Configurations*
> > Laptop with i7 processor and 16 GB RAM.
> > TPCH Data Scale: 100 GB
> > No Sort with no inverted index data.
> > Total CarbonData Size : 32 GB
> > Total Parquet Size : 31 GB
> >
> >
> > Queries Parquet Carbon New Carbon Old Carbon Old vs Carbon New Carbon New
> > Vs Parquet Carbon old Vs Parquet
> > Q1 101 96 128 25.00% 4.95% -26.73%
> > Q2 85 82 85 3.53% 3.53% 0.00%
> > Q3 118 112 135 17.04% 5.08% -14.41%
> > Q4 473 424 486 12.76% 10.36% -2.75%
> > Q5 228 201 205 1.95% 11.84% 10.09%
> > Q6 19.2 19.2 48 60.00% 0.00% -150.00%
> > Q7 194 181 198 8.59% 6.70% -2.06%
> > Q8 285 263 275 4.36% 7.72% 3.51%
> > Q9 362 345 363 4.96% 4.70% -0.28%
> > Q10 101 92 93 1.08% 8.91% 7.92%
> > Q11 64 61 62 1.61% 4.69% 3.13%
> > Q12 41.4 44 63 30.16% -6.28% -52.17%
> > Q13 43.4 43.6 43.7 0.23% -0.46% -0.69%
> > Q14 36.9 31.5 41 23.17% 14.63% -11.11%
> > Q15 70 59 80 26.25% 15.71% -14.29%
> > Q16 64 60 64 6.25% 6.25% 0.00%
> > Q17 426 418 432 3.24% 1.88% -1.41%
> > Q18 1015 921 1001 7.99% 9.26% 1.38%
> > Q19 62 53 59 10.17% 14.52% 4.84%
> > Q20 406 326 426 23.47% 19.70% -4.93%
> > Full Scan Query* 140 116 164 29.27% 17.14% -17.14%
> > *Full Scan Query means count of every coumn of lineitem, In this way we
> can
> > check the full scan query performance.
> >
> > The above optimization is not just limited to fileformat and Presto
> > integration but also improves for CarbonSession integration.
> > We can further optimize carbon by the tasks(Vishal is already working on
> > it) like adaptive encoding for all types of columns and storing length
> and
> > values in separate pages in case of string datatype.Please refer
> >
>
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Carbondata-Store-size-optimization-td62283.html> > .
> >
> > --
> > Thanks & Regards,
> > Ravi
> >
>
>
>
>