[Discussion] DDLs to operate on CarbonLRUCache

Posted by Naman Rastogi on Feb 18, 2019; 10:01am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Discussion-DDLs-to-operate-on-CarbonLRUCache-tp75197.html

Hi all,

Currently carbon supports caching mechanism for Blocks/Blocklets. Even
though it allows end user to set the Cache size, it is still very
limited in functionality, and user arbitrarily chooses the carbon
property *carbon.max.driver.lru.cache.size* where before launching the
carbon session, he/she has no idea of how much cache should be set for
his/her requirement.

For this problem, I propose the following imporvements in carbon caching
mechanism.

1. Support DDL for showing current cache used per table.
2. Support DDL for showing current cache used for a particular table.
    For these two points, QiangCai has already has a PR:
    https://github.com/apache/carbondata/pull/3078

3. Support DDL for clearing all the entries cache.
    This will look like:
    CLEAN CACHE

4. Support DDL for clearing cache for a particular table.
    This will clear all the entries in the cache which belong to a
    particular table. This will look like
    CLEAN CACHE FOR TABLE tablename

5. Support DDL to estimate required cache for a particular table.
    As explained above, the user does not know beforehand how much cache
    will be required for his/her current work. So this DDL will let the
    user estimate how much cache will be required for a particular
    table. For this we will launch a job and estimate the memory
    required for all the blocks, and sum it up.

6. Dynamic "max cache size" configration
    Suppose now the user knows required cache size he needs, but the
    current system requires the user to set the
    *carbon.max.driver.lru.cache.size* and restart the JDBC server for
    it to take effect. For this I am suggesting to make the carbon
    property *carbon.max.driver.lru.cache.size* dynamically configurable
    which allows the user to change the max LRU cache size on the fly.

Any suggestion from the community is greatly appreciated.

Thanks

Regards

Naman Rastogi
Technical Lead - BigData Kernel
Huawei Technologies India Pvt. Ltd.