Re: [Discussion] DDLs to operate on CarbonLRUCache

Posted by Naman Rastogi on Feb 20, 2019; 2:18pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Discussion-DDLs-to-operate-on-CarbonLRUCache-tp75197p75269.html

@akash

1. No sync issues will be there are the operation will be atomic, only
one operation is allowed on lruCache at a particular point of time (will
be taken care in code).

2. It will act as it is expected to, concurrent operation on cache will
be blocked.

3 & 4. We will do it the same way we do now. Summing up memory size for
all the BlockDataMap -> summing up for all the DataMapRow.
Just that we will distribute that among different executors.

Regards

Naman Rastogi
Technical Lead - BigData Kernel
Huawei Technologies India Pvt. Ltd.


On Wed, Feb 20, 2019 at 7:51 PM Naman Rastogi <[hidden email]>
wrote:

> @dhatchayani
>
> If we are to support estimation of cache size required for a particular
> table, we have to make max_cache_size dynamic, and change the cache size
> dynamically, otherwise user will have to restart the JDBC server all
> over again, which does not seem like a good idea from the perspective of
> end user. So, it will be like one default system property will be there
> (CARBON.MAX.DRIVER.LRU.CACHE.SIZE) which will be considered while
> starting the server, and then user, later, can change the size of cache
> according to his need on the fly using session property
> CARBON.MAX.DRIVER.LRU.CACHE.SIZE.DYNAMIC (or something else) using set
> command
> set CARBON.MAX.DRIVER.LRU.CACHE.SIZE.DYNAMIC=100
> here 100 is in MB, which is similar to CARBON.MAX.DRIVER.LRU.CACHE.SIZE.
>
> One restriction we can add over this is that user, while changing the
> size of cache, can only increase the cache size, and not decrease it, so
> that we dont have to the the remove some entries which are already
> present in the cache. This is also open to discussion, that we should
> add this restriction or not.
>
> So, this way, user will be able to start the server with some value for
> max cache size, can be -1 as well, then change the max cache size on the
> fly.
> Regards
>
> Naman Rastogi
> Technical Lead - BigData Kernel
> Huawei Technologies India Pvt. Ltd.
>
>
> On Tue, Feb 19, 2019 at 6:51 PM akashrn5 <[hidden email]> wrote:
>
>> Hi naman,
>>
>> Thanks for proposing the feature. Looks really helpful from user and
>> developer perspective.
>>
>> Basically needed the design document, so that all the doubts would be
>> cleared.
>>
>> 1. Basicaly how you are going to handle the sync issues like, multiple
>> queries with drop and show cache. are you going to introduce any locking
>> mechanism?
>>
>> 2. if user clears cache during query? how it is noing to behave? is it
>> allowed or concurrent operaion is blocked?
>>
>> 3. How it os gonna work with Distributed index server, and types of it
>> like
>> embedded, presto and other and local server? basically what is the impact
>> with that?
>>
>> 4. You said you will launch a job to get the size from all the blocks
>> present. Currently we create the block or blocklet datamap, and calcute
>> each
>> datamap size and then we add to cache based on the lru cach esize
>> configured. So wanted to know how you will be calculating the size in your
>> case
>>
>> Regards,
>> Akash
>>
>>
>>
>> --
>> Sent from:
>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>>
>