Posted by
Naman Rastogi on
Feb 20, 2019; 2:18pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Discussion-DDLs-to-operate-on-CarbonLRUCache-tp75197p75268.html
@dhatchayani
If we are to support estimation of cache size required for a particular
table, we have to make max_cache_size dynamic, and change the cache size
dynamically, otherwise user will have to restart the JDBC server all
over again, which does not seem like a good idea from the perspective of
end user. So, it will be like one default system property will be there
(CARBON.MAX.DRIVER.LRU.CACHE.SIZE) which will be considered while
starting the server, and then user, later, can change the size of cache
according to his need on the fly using session property
CARBON.MAX.DRIVER.LRU.CACHE.SIZE.DYNAMIC (or something else) using set
command
set CARBON.MAX.DRIVER.LRU.CACHE.SIZE.DYNAMIC=100
here 100 is in MB, which is similar to CARBON.MAX.DRIVER.LRU.CACHE.SIZE.
One restriction we can add over this is that user, while changing the
size of cache, can only increase the cache size, and not decrease it, so
that we dont have to the the remove some entries which are already
present in the cache. This is also open to discussion, that we should
add this restriction or not.
So, this way, user will be able to start the server with some value for
max cache size, can be -1 as well, then change the max cache size on the
fly.
Regards
Naman Rastogi
Technical Lead - BigData Kernel
Huawei Technologies India Pvt. Ltd.
On Tue, Feb 19, 2019 at 6:51 PM akashrn5 <
[hidden email]> wrote:
> Hi naman,
>
> Thanks for proposing the feature. Looks really helpful from user and
> developer perspective.
>
> Basically needed the design document, so that all the doubts would be
> cleared.
>
> 1. Basicaly how you are going to handle the sync issues like, multiple
> queries with drop and show cache. are you going to introduce any locking
> mechanism?
>
> 2. if user clears cache during query? how it is noing to behave? is it
> allowed or concurrent operaion is blocked?
>
> 3. How it os gonna work with Distributed index server, and types of it like
> embedded, presto and other and local server? basically what is the impact
> with that?
>
> 4. You said you will launch a job to get the size from all the blocks
> present. Currently we create the block or blocklet datamap, and calcute
> each
> datamap size and then we add to cache based on the lru cach esize
> configured. So wanted to know how you will be calculating the size in your
> case
>
> Regards,
> Akash
>
>
>
> --
> Sent from:
>
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/>