[jira] [Resolved] (CARBONDATA-2246) Fix not-enough-memory bugs in unsafe data loading

[Akash R Nilugal (Jira)]

     [ https://issues.apache.org/jira/browse/CARBONDATA-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

xuchuanyin resolved CARBONDATA-2246.
------------------------------------
    Resolution: Fixed

> Fix not-enough-memory bugs in unsafe data loading
> -------------------------------------------------
>
>                 Key: CARBONDATA-2246
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-2246
>             Project: CarbonData
>          Issue Type: Bug
>            Reporter: xuchuanyin
>            Assignee: xuchuanyin
>            Priority: Major
>          Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Currently in carbon data loading, if we enable the unsafe loading and specify corresponding properties, data loading will end in OOM.
> The key properties to reproduce the bug are as following:
> ```
> 01: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_INMEMORY_MERGE_SORT, "true") CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_UNSAFE_SORT, "true")
>  02:
>  03: // unsafe sort memory manager
>  04: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.IN_MEMORY_STORAGE_FOR_SORTED_DATA_IN_MB, "1024")
>  05:
>  06: // unsafe working memory manager
>  07: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB, "512")
>  08:
>  09: // one unsafe page, better if loading_cores * this < memory
>  10: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.OFFHEAP_SORT_CHUNK_SIZE_IN_MB, "512")
> ```
>  
> Notice that the `OFFHEAP_SORT_CHUNK_SIZE_IN_MB` are exactly the same as `UNSAFE_WORKING_MEMORY_IN_MB` which will cause problem

--
This message was sent by Atlassian JIRA
(v7.6.3#76005)