[jira] [Created] (CARBONDATA-2889) Support Decoder based fall back mechanism in Local Dictionary

[Akash R Nilugal (Jira)]

Akash R Nilugal created CARBONDATA-2889:
-------------------------------------------

             Summary: Support Decoder based fall back mechanism in Local Dictionary
                 Key: CARBONDATA-2889
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-2889
             Project: CarbonData
          Issue Type: Sub-task
            Reporter: Akash R Nilugal
            Assignee: Akash R Nilugal


Currently, when the fallback is initiated for a column page in case of local dictionary, we are keeping both encoded data

and actual data in memory and then we form the new column page without dictionary encoding and then at last we free the Encoded Column Page.

Because of this offheap memory footprint increases.

 

We can reduce the offheap memory footprint. This can be done using decoder based fallback mechanism.

This means, no need to keep the actual data along with encoded data in encoded column page. We can keep only encoded data and to form a new column page, get the dictionary data from encoded column page by uncompressing and using dictionary data get the actual data using local dictionary generator and put it in new column page created and compress it again and give to consumer for writing blocklet. 

 

The above process may slow down the loading, but it will reduces the memory footprint. So we can give a property which will decide whether to take current fallback procedure or decoder based fallback mechanism dring fallback



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)