Login  Register

Re: [DISCUSSION] About data backward compatibility

Posted by bill.zhou on Aug 16, 2017; 12:18am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/DISCUSSION-About-data-backward-compatibility-tp20183p20229.html

hi  Jacky & Ravindra
 
 I think it should make sure the migration tool performance is good then can chose option 2.
like some user already use v1 for 1 year, there are 1 year history data, the Migration tool should be make sure the high performance. following is my thinking the key feature should have
1. the tool should have high performance, like 1 year data can migration in 1 day.
2. the tool support breakpoint migration, like there is 1 one table which has 1 year data and 365 segment, user can migrate this table one by one segment, and can read this table for the segment which already migrated .
ravipesala wrote
Hi Jacky,

I feel option 2 is better but it should have one time migration of data
from old formats to latest format. So first we should have migration tool
support to read old format data using the old version and write using the
latest version. This migration should be capable of supporting any old
format to latest format.  We can support only V3 going forward so we can
clean all the old format code after having this tool.

Regards,
Ravindra.

On 12 August 2017 at 11:43, Jacky Li <[hidden email]> wrote:

> Hi All,
>
> As I am implementing new encoding feature for carbondata, I found it is
> hard to maintain both read and write backward compatibility with all
> CarbonData format including V1, V2, and V3.
>
> In this post, I want to discuss the roadmap for backward compatibility
> support.
>
> I am proposing following feature plan:
> 1. For the write support. Start from CarbonData 1.2 onwards, support
> writing V3 format only.
> V3 format is introduced in CarbonData 1.1 (2017 Feb), and it is stable for
> more than half year now. And since we are going to add new feature in V3
> format only, it is better we clean the writing path for V3 format. If there
> are bugs in V1 and V2 format, we still will fix it in maintenance version
> before CarbonData 1.1
>
> 2. For the read support, there are two options.
> Option 1: Support reading V1 and V2 format, and in CarbonData 1.3, build
> data migration tool to help user to migrate old carbon store. Stop
> supporting reading V1 and V2 after CarbonData 1.3
> The pro is that if there are still some users are using V1 or V2 carbon in
> there application, they can continue to use CarbonData 1.2 to read the old
> data.
> The con is that any new feature introduced for V3 need to be careful and
> should not break read compatibility of V1 and V2. Like, some new encoding
> will be every hard to introduce.
>
> Option 2: Support reading V3 format starting from CarbonData 1.2
> The pro is that code will be more clean and no restriction of add new
> encoding.
> The con is that any old carbon store that based on V1 and V2 format, it
> can be read using CarbonData 1.1 only.
>
> I want to collect the opinion form community, if there are users still
> using V1 or V2 format, I think it is saver to go with Option 1. Otherwise,
> if all users are using V3 format (CarbonData 1.1 and 1.1.1), I think Option
> 2 is a better choice.
>
>
> Thanks,
> Jacky Li
>
>


--
Thanks & Regards,
Ravi