[jira] [Updated] (CARBONDATA-2246) Fix out-of-memory bugs in unsafe data loading

[Akash R Nilugal (Jira)]

     [ https://issues.apache.org/jira/browse/CARBONDATA-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

xuchuanyin updated CARBONDATA-2246:
-----------------------------------
    Description:
Currently in carbon data loading, if we enable the unsafe loading and specify corresponding properties, data loading will end in OOM.

The key properties to reproduce the bug are as following:

```

01: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_INMEMORY_MERGE_SORT, "true") CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_UNSAFE_SORT, "true")
 02:
 03: // unsafe sort memory manager
 04: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.IN_MEMORY_STORAGE_FOR_SORTED_DATA_IN_MB, "1024")
 05:
 06: // unsafe working memory manager
 07: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB, "512")
 08:
 09: // one unsafe page, better if loading_cores * this < memory
 10: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.OFFHEAP_SORT_CHUNK_SIZE_IN_MB, "512")

```

 

Notice that the `OFFHEAP_SORT_CHUNK_SIZE_IN_MB` are exactly the same as `UNSAFE_WORKING_MEMORY_IN_MB` which will cause problem

  was:
Currently in carbon data loading, if we enable the unsafe loading and specify corresponding properties, data loading will end in OOM.

The key properties to reproduce the bug are as following:

```

01: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_INMEMORY_MERGE_SORT, "true") CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_UNSAFE_SORT, "true")
02:
03: // unsafe sort memory manager
04: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.IN_MEMORY_STORAGE_FOR_SORTED_DATA_IN_MB, "1024")
05:
06: // unsafe working memory manager
07: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB, "512")
08:
09: // one unsafe page, better if loading_cores * this < memory
10: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.OFFHEAP_SORT_CHUNK_SIZE_IN_MB, "512")

```

> Fix out-of-memory bugs in unsafe data loading
> ---------------------------------------------
>
>                 Key: CARBONDATA-2246
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-2246
>             Project: CarbonData
>          Issue Type: Bug
>            Reporter: xuchuanyin
>            Assignee: xuchuanyin
>            Priority: Major
>
> Currently in carbon data loading, if we enable the unsafe loading and specify corresponding properties, data loading will end in OOM.
> The key properties to reproduce the bug are as following:
> ```
> 01: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_INMEMORY_MERGE_SORT, "true") CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_UNSAFE_SORT, "true")
>  02:
>  03: // unsafe sort memory manager
>  04: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.IN_MEMORY_STORAGE_FOR_SORTED_DATA_IN_MB, "1024")
>  05:
>  06: // unsafe working memory manager
>  07: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB, "512")
>  08:
>  09: // one unsafe page, better if loading_cores * this < memory
>  10: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.OFFHEAP_SORT_CHUNK_SIZE_IN_MB, "512")
> ```
>  
> Notice that the `OFFHEAP_SORT_CHUNK_SIZE_IN_MB` are exactly the same as `UNSAFE_WORKING_MEMORY_IN_MB` which will cause problem

--
This message was sent by Atlassian JIRA
(v7.6.3#76005)