[jira] [Resolved] (CARBONDATA-2927) Multiple issue fixes for varchar column and complex columns that grows more than 2MB

[Akash R Nilugal (Jira)]

     [ https://issues.apache.org/jira/browse/CARBONDATA-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravindra Pesala resolved CARBONDATA-2927.
-----------------------------------------
       Resolution: Fixed
    Fix Version/s: 1.5.0

> Multiple issue fixes for varchar column and complex columns that grows more than 2MB
> ------------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-2927
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-2927
>             Project: CarbonData
>          Issue Type: Bug
>            Reporter: Ajantha Bhat
>            Assignee: Ajantha Bhat
>            Priority: Major
>             Fix For: 1.5.0
>
>          Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> *Fixed:*
>  *1. varchar data length is more than 2MB, buffer overflow exception (thread local row buffer)*
> *root* casue*: thread* loaclbuffer *was hardcoded with 2MB.*
> *solution: grow dynamically based on the row size.*
>  *2. read data from carbon file having one row of varchar data with 150 MB length is very slow.*
> *root casue:  At UnsafeDMStore, ensure memory is just incresing by 8KB each time and lot of time malloc and free happens before reaching 150MB. hence very slow performance.*
> *solution: directly check and allocate the required size.*
>  *3. Jvm crash when data size is more than 128 MB in unsafe sort step.*
> *root cause: unsafeCarbonRowPage is of 128MB, so if data is more than 128MB  for one row, we access block beyond allocated, leading to JVM crash.*
> *solution: validate the size before access and prompt user to increase unsafe memory. (by carbon property)*

--
This message was sent by Atlassian JIRA
(v7.6.3#76005)