[jira] [Resolved] (CARBONDATA-3519) Optimizations in write step to avoid unnecessary memory blk allocation/free

[Akash R Nilugal (Jira)]

     [ https://issues.apache.org/jira/browse/CARBONDATA-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jacky Li resolved CARBONDATA-3519.
----------------------------------
    Fix Version/s: 2.0.0
       Resolution: Fixed

> Optimizations in write step to avoid unnecessary memory blk allocation/free
> ---------------------------------------------------------------------------
>
>                 Key: CARBONDATA-3519
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-3519
>             Project: CarbonData
>          Issue Type: Improvement
>          Components: core
>            Reporter: Venugopal Reddy K
>            Priority: Minor
>             Fix For: 2.0.0
>
>          Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
>  +*Issue-1:*+
> {color:#0747a6}*Context:*{color}
> For a string column with local dictionary enabled, a column page of
> `{color:#de350b}UnsafeFixLengthColumnPage{color}` with datatype `{color:#de350b}{{DataTypes.BYTE_ARRAY}}{color}` is created for `{color:#de350b}{{encodedPage}}{color}` along with regular `{color:#de350b}{{actualPage}}{color}` of `{color:#de350b}{{UnsafeVarLengthColumnPage}}{color}`. 
> We have `{color:#de350b}*{{capacity}}*{color}` field in the `{color:#de350b}{{UnsafeFixLengthColumnPage}}{color}`. And this field indicates the capacity of  allocated
> `{color:#de350b}{{memoryBlock}}{color}` for the page. `{{{color:#de350b}ensureMemory{color}()}}` method gets called while adding rows to check if  `{color:#de350b}{{totalLength + requestSize > capacity}}{color}` to allocate a new memoryBlock. If there is no room to add the next row, allocates a new block, copy the old context(prev rows) and free the old memoryBlock.
> {color:#0747a6} *Problem:*{color}
> While, `{color:#de350b}{{UnsafeFixLengthColumnPage}}{color}` with with datatype `{color:#de350b}{{DataTypes.BYTE_ARRAY}}{color}` is created for `{color:#de350b}{{encodedPage}}{color}`, we have not assigned the *`{color:#de350b}{{capacity}}{color}`* field with allocated memory block size. Hence, for each add row to tablePage, *ensureMemory() check always fails*, allocates a new column page memoryBlock, copy the old context(prev rows) and free the old memoryBlock. This *allocation of new memoryBlock and free of old memoryBlock happens for each row row addition* for the string columns with local dictionary.
>  
> +*Issue-2:*+
> {color:#0747a6}*Context:*{color}
> In`{color:#de350b}VarLengthColumnPageBase{color}`, we have a `{color:#de350b}rowOffset{color}` column page of  `{color:#de350b}UnsafeFixLengthColumnPage{color}` of datatype `{color:#de350b}INT{color}`
> to maintain the data offset to {color:#172b4d}each{color} row of variable length columns. This `{color:#de350b}rowOffset{color}` page allocates to be size of page. 
> {color:#0747a6} *Problem:*{color}
> {color:#172b4d}If we have 10 rows in the page, we need 11 rows for its rowOffset page. Because we always keep 0 as offset to 1st row. So an additional row is required for rowOffset page[pasted code below to show the reference]. Otherwise, *ensureMemory() check always fails for the last row*(10th row in this case) of data and *allocates a new rowOffset page memoryBlock, copy the old context(prev rows) and free the old memoryBlock*. This *can happen for the string columns with local dictionary, direct dictionary columns, global disctionary columns*.{color}
>  
> {code:java}
> public abstract class VarLengthColumnPageBase extends ColumnPage {
> ...
> @Override
> public void putBytes(int rowId, byte[] bytes) {
>  ...
>  if (rowId == 0) {
>  rowOffset.putInt(0, 0); ==> offset to 1st row is 0.
>  }
>  rowOffset.putInt(rowId + 1, rowOffset.getInt(rowId) + bytes.length);
>  putBytesAtRow(rowId, bytes);
>  totalLength += bytes.length;
> }
> ...
> }
>  
> {code}
>  

--
This message was sent by Atlassian Jira
(v8.3.4#803005)