This post was updated on .
I did an experiment these two days.Insert thousands of batches of data into a
table use global_sort.I found that the first query use 4800s JDBC memery is 14GB;when first finish i do the second query immediately use the same sql "select count(*) from sameTable". The second query take 3700s and the JDBC memery increase to 20GB,between first and second there is no other insertion and query.see the LRU debug log we can see like this:" Removed entry from InMemory lru cache :: hdfs://hacluster/user/hive/warehouse/carbon.store/x/xx_17910/Fact/Part0/Segment_2497/2497_1547575104816.carbonindexmerge | org.apache.carbondata.core.cache.CarbonLRUCache.removeKey(CarbonLRUCache.java:163)2" maybe the LRU cache is remove from memery,beside there is not another query. From these phenomena, it may be seen that there are several problems. 1)is there has memory leaks for the 14GB ->20GB memery use.(i have jmap dump file) 2)Can the logic of LRU be optimized? 3)Is it possible to concurrently load index Files to LRU Using thread pools -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ |
Do you mean for the same query, even if the memory configured for LRU is big enough, some of the cached entries for index still get purged?
If it is so, then there maybe some problems with the LRU cache I think. However, since your query is purely count-star without any conditions, I suppose the query may not use the index are all, which means the LRU cache may donot work for your query. From Smartisan T1 |
In reply to this post by litao
Do you mean for the same query, even if the memory configured for LRU is big enough, some of the cached entries for index still get purged?
If it is so, then there maybe some problems with the LRU cache I think. However, since your query is purely count-star without any conditions, I suppose the query may not use the index are all, which means the LRU cache may donot work for your query. From Smartisan T1 |
Do you mean for the same query, even if the memory configured for LRU is big
enough, some of the cached entries for index still get purged? --yes i think it is. However, since your query is purely count-star without any conditions, I suppose the query may not use the index are all, which means the LRU cache may donot work for your query. --but by logs we can see the carbonindexmerge file load to LRUCache. -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ |
@xuchuanyin
Count star on carbon rely over the carbon index and carbonindexmerge so it will load the index files into the driver LRU cache. @litao If there is no other query and executing the same sql again then increasing memory footprint from 14g to 20g really looks like a memory leak. -Regards, Shahid On Thu 17 Jan, 2019, 6:29 AM litao <[hidden email] wrote: > Do you mean for the same query, even if the memory configured for LRU is > big > enough, some of the cached entries for index still get purged? > --yes i think it is. > > However, since your query is purely count-star without any conditions, I > suppose the query may not use the index are all, which means the LRU cache > may donot work for your query. > --but by logs we can see the carbonindexmerge file load to LRUCache. > > > > -- > Sent from: > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ > |
Free forum by Nabble | Edit this page |