[Discussion] Abstracting CarbonData's Index Interface
Locked
 
|
This post was updated on .
Hi community,
Currently CarbonData has builtin index support which is one of its key strength. Using index, CarbonData can perform very fast filter query by filtering on blocks and blocklets. However, it also introduces some cost like memory consumption of the index tree and impact first query time because the process of loading of index from file footer into memory. On the other hand, user way want to deploy CarbonData in a multi-tennant environment, in which case multiple applications may access data files simultaneously, which again exacerbate this resource consumption issue. So, I want to propose and discuss a solution with you to solve this problem and make an abstraction of interface for CarbonData's future evolvement. I am thinking the final result of this work should achieve at least two goals: Goal 1: User can choose the place to store Index data, it can be stored in processing framework's memory space (like in spark driver memory) or in another service outside of the processing framework (like using a independent database service). These indices can be shared across applications in a scalable way. Goal 2: Developer can add more index of his choice to CarbonData files. Besides B+ tree on multi-dimensional key which current CarbonData supports, developers are free to add other indexing technology to make certain workload faster. These new indices should be added in a pluggable way. In order to achieve these goals, an abstraction need to be created for CarbonData project, including: - Segment: each segment is presenting one load of data, and tie with some indices created with this load - Index: index is created when this segment is created, and is leveraged when CarbonInputFormat's getSplit is called, to filter out the required blocks or even blocklets. - CarbonInputFormat: There maybe n number of indices created for data file, when querying these data files, InputFormat should know how to access these indices, and initialize or loading these index if required. Obviously, this work should be separated into different tasks and implemented gradually. But first of all, let's discuss on the goal and the proposed approach. What is your idea? Regards, Jacky |
Re: Abstracting CarbonData's Index Interface
|
|
Yes Jacky, interfaces needs to be revisited.
For Goal 1 and Goal 2: abstraction required for both Index and Index store. Also multi-column index(composite index) needs to be considered. Regards, Ramana On Sat, Oct 1, 2016 at 11:01 AM, Jacky Li <[hidden email]> wrote: > Hi community, > > Currently CarbonData have builtin index support which is one of the key > strength of CarbonData. Using index, CarbonData can do very fast filter > query by filtering on block and blocklet level. However, it also introduces > memory consumption of the index tree and impact first query time because > the > process of loading of index from file footer into memory. On the other > side, > in a multi-tennant environment, multiple applications may access data files > simultaneously, which again exacerbate this resource consumption issue. > So, I want to propose and discuss a solution with you to solve this > problem and make an abstraction of interface for CarbonData's future > evolvement. > I am thinking the final result of this work should achieve at least two > goals: > > Goal 1: User can choose the place to store Index data, it can be stored in > processing framework's memory space (like in spark driver memory) or in > another service outside of the processing framework (like using a > independent database service) > > Goal 2: Developer can add more index of his choice to CarbonData files. > Besides B+ tree on multi-dimensional key which current CarbonData supports, > developers are free to add other indexing technology to make certain > workload faster. These new indices should be added in a pluggable way. > > In order to achieve these goals, an abstraction need to be created for > CarbonData project, including: > > - Segment: each segment is presenting one load of data, and tie with some > indices created with this load > > - Index: index is created when this segment is created, and is leveraged > when CarbonInputFormat's getSplit is called, to filter out the required > blocks or even blocklets. > > - CarbonInputFormat: There maybe n number of indices created for data file, > when querying these data files, InputFormat should know how to access these > indices, and initialize or loading these index if required. > > Obviously, this work should be separated into different tasks and > implemented gradually. But first of all, let's discuss on the goal and the > proposed approach. What is your idea? > > Regards, > Jacky > > > > > > -- > View this message in context: http://apache-carbondata- > mailing-list-archive.1130556.n5.nabble.com/Abstracting- > CarbonData-s-Index-Interface-tp1587.html > Sent from the Apache CarbonData Mailing List archive mailing list archive > at Nabble.com. > |
|
|
I am currently thinking these abstractions:
- A SegmentManager is the global manager of all segments for one table. It can be used to get all segments and manage the segment while loading and compaction. - A CarbonInputFormat will take the input of table path, so means it represent the whole table contain all segments. When getSplit is called, it will get all segments by calling SegmentManager interface. - Each Segment contains a list of Index, and an IndexSelector. While currently CarbonData only has MDK index, developer can create multiple indices for each segment in the future. - An Index is an interface to filtering on block/blocklet, and provide this functionality only. Implementation should hide all complexity like deciding where to store the index. - An IndexSelector is an interface to choose which index to use based on query predicates. Default implementation is to choose the first index. An implementation of IndexChooser can also decide not to use index at all. - A Distributor is used to map the filtered block/blocklet to InputSplits. Implementation can take number of node, parallelism into consideration. It can also decide to distribute tasks based on block or blocklet. So the main concepts are SegmentManager, Segment, Index, IndexSelector, InputFormat/OutputFormat, Distributor. There will be a default implementation of CarbonInputFormat whose getSplit will do the following: 1. gat all segments by calling SegmentManager 2. for each segment, choose the index to use by IndexSelector 3. invoke the selected Index to filter out block/blocklet (since these are two concept, maybe a parent class need to be created to encapsulate them) 4. distribute the filtered block/blocklet to InputSplits by Distributor. Regarding the input to the Index.filter interface, I have not decided to use the existing QueryModel or create a new cleaner QueryModel interface. If new QueryModel is desired, it should only contain filter predicate and project columns, so it is much simpler than current QueryModel. But I see current QueryModel is used in Compaction also, so I think it is better to do this clean up later? Does this look fine to you? Any suggestion is welcome. Regards, Jacky > 在 2016年10月3日，上午2:18，Venkata Gollamudi <[hidden email]> 写道： > > Yes Jacky, interfaces needs to be revisited. > For Goal 1 and Goal 2: abstraction required for both Index and Index store. > Also multi-column index(composite index) needs to be considered. > > Regards, > Ramana > > On Sat, Oct 1, 2016 at 11:01 AM, Jacky Li <[hidden email]> wrote: > >> Hi community, >> >> Currently CarbonData have builtin index support which is one of the key >> strength of CarbonData. Using index, CarbonData can do very fast filter >> query by filtering on block and blocklet level. However, it also introduces >> memory consumption of the index tree and impact first query time because >> the >> process of loading of index from file footer into memory. On the other >> side, >> in a multi-tennant environment, multiple applications may access data files >> simultaneously, which again exacerbate this resource consumption issue. >> So, I want to propose and discuss a solution with you to solve this >> problem and make an abstraction of interface for CarbonData's future >> evolvement. >> I am thinking the final result of this work should achieve at least two >> goals: >> >> Goal 1: User can choose the place to store Index data, it can be stored in >> processing framework's memory space (like in spark driver memory) or in >> another service outside of the processing framework (like using a >> independent database service) >> >> Goal 2: Developer can add more index of his choice to CarbonData files. >> Besides B+ tree on multi-dimensional key which current CarbonData supports, >> developers are free to add other indexing technology to make certain >> workload faster. These new indices should be added in a pluggable way. >> >> In order to achieve these goals, an abstraction need to be created for >> CarbonData project, including: >> >> - Segment: each segment is presenting one load of data, and tie with some >> indices created with this load >> >> - Index: index is created when this segment is created, and is leveraged >> when CarbonInputFormat's getSplit is called, to filter out the required >> blocks or even blocklets. >> >> - CarbonInputFormat: There maybe n number of indices created for data file, >> when querying these data files, InputFormat should know how to access these >> indices, and initialize or loading these index if required. >> >> Obviously, this work should be separated into different tasks and >> implemented gradually. But first of all, let's discuss on the goal and the >> proposed approach. What is your idea? >> >> Regards, >> Jacky >> >> >> >> >> >> -- >> View this message in context: http://apache-carbondata- >> mailing-list-archive.1130556.n5.nabble.com/Abstracting- >> CarbonData-s-Index-Interface-tp1587.html >> Sent from the Apache CarbonData Mailing List archive mailing list archive >> at Nabble.com. >> |
|
|
After a second thought regarding the index part, another option is that to have a very simple Segment definition which can only list all files it has or listFile taking the QueryModel as input, implementation of Segment can be IndexSegment, MultiIndexSegment or StreamingSegment (no index). In future, developer is free to create MultiIndexSegment to select index internally. Is this option better?
Regards, Jacky > 在 2016年10月3日，上午11:00，Jacky Li <[hidden email]> 写道： > > I am currently thinking these abstractions: > > - A SegmentManager is the global manager of all segments for one table. It can be used to get all segments and manage the segment while loading and compaction. > - A CarbonInputFormat will take the input of table path, so means it represent the whole table contain all segments. When getSplit is called, it will get all segments by calling SegmentManager interface. > - Each Segment contains a list of Index, and an IndexSelector. While currently CarbonData only has MDK index, developer can create multiple indices for each segment in the future. > - An Index is an interface to filtering on block/blocklet, and provide this functionality only. Implementation should hide all complexity like deciding where to store the index. > - An IndexSelector is an interface to choose which index to use based on query predicates. Default implementation is to choose the first index. An implementation of IndexChooser can also decide not to use index at all. > - A Distributor is used to map the filtered block/blocklet to InputSplits. Implementation can take number of node, parallelism into consideration. It can also decide to distribute tasks based on block or blocklet. > > So the main concepts are SegmentManager, Segment, Index, IndexSelector, InputFormat/OutputFormat, Distributor. > > There will be a default implementation of CarbonInputFormat whose getSplit will do the following: > 1. gat all segments by calling SegmentManager > 2. for each segment, choose the index to use by IndexSelector > 3. invoke the selected Index to filter out block/blocklet (since these are two concept, maybe a parent class need to be created to encapsulate them) > 4. distribute the filtered block/blocklet to InputSplits by Distributor. > > Regarding the input to the Index.filter interface, I have not decided to use the existing QueryModel or create a new cleaner QueryModel interface. If new QueryModel is desired, it should only contain filter predicate and project columns, so it is much simpler than current QueryModel. But I see current QueryModel is used in Compaction also, so I think it is better to do this clean up later? > > > Does this look fine to you? Any suggestion is welcome. > > Regards, > Jacky > > >> 在 2016年10月3日，上午2:18，Venkata Gollamudi <[hidden email]> 写道： >> >> Yes Jacky, interfaces needs to be revisited. >> For Goal 1 and Goal 2: abstraction required for both Index and Index store. >> Also multi-column index(composite index) needs to be considered. >> >> Regards, >> Ramana >> >> On Sat, Oct 1, 2016 at 11:01 AM, Jacky Li <[hidden email]> wrote: >> >>> Hi community, >>> >>> Currently CarbonData have builtin index support which is one of the key >>> strength of CarbonData. Using index, CarbonData can do very fast filter >>> query by filtering on block and blocklet level. However, it also introduces >>> memory consumption of the index tree and impact first query time because >>> the >>> process of loading of index from file footer into memory. On the other >>> side, >>> in a multi-tennant environment, multiple applications may access data files >>> simultaneously, which again exacerbate this resource consumption issue. >>> So, I want to propose and discuss a solution with you to solve this >>> problem and make an abstraction of interface for CarbonData's future >>> evolvement. >>> I am thinking the final result of this work should achieve at least two >>> goals: >>> >>> Goal 1: User can choose the place to store Index data, it can be stored in >>> processing framework's memory space (like in spark driver memory) or in >>> another service outside of the processing framework (like using a >>> independent database service) >>> >>> Goal 2: Developer can add more index of his choice to CarbonData files. >>> Besides B+ tree on multi-dimensional key which current CarbonData supports, >>> developers are free to add other indexing technology to make certain >>> workload faster. These new indices should be added in a pluggable way. >>> >>> In order to achieve these goals, an abstraction need to be created for >>> CarbonData project, including: >>> >>> - Segment: each segment is presenting one load of data, and tie with some >>> indices created with this load >>> >>> - Index: index is created when this segment is created, and is leveraged >>> when CarbonInputFormat's getSplit is called, to filter out the required >>> blocks or even blocklets. >>> >>> - CarbonInputFormat: There maybe n number of indices created for data file, >>> when querying these data files, InputFormat should know how to access these >>> indices, and initialize or loading these index if required. >>> >>> Obviously, this work should be separated into different tasks and >>> implemented gradually. But first of all, let's discuss on the goal and the >>> proposed approach. What is your idea? >>> >>> Regards, >>> Jacky >>> >>> >>> >>> >>> >>> -- >>> View this message in context: http://apache-carbondata- >>> mailing-list-archive.1130556.n5.nabble.com/Abstracting- >>> CarbonData-s-Index-Interface-tp1587.html >>> Sent from the Apache CarbonData Mailing List archive mailing list archive >>> at Nabble.com. >>> > |
Re: Abstracting CarbonData's Index Interface
|
|
I would agree with having simple segment definition. Segment can use a
metadata info that describes the segment - For example; Segment type, index availability, index type, index storage type (attached or detached/secondary) etc. For streaming ingest segment, it also may possibly contain min-max kind of information for each blocklet, that can used for indexing. So implementation details of different segment types can be hidden from user. We may have to think about partitioning support along with load segments in future. Best Regards, Aniket On Sun, Oct 2, 2016 at 10:25 PM, Jacky Li <[hidden email]> wrote: > After a second thought regarding the index part, another option is that to > have a very simple Segment definition which can only list all files it has > or listFile taking the QueryModel as input, implementation of Segment can > be IndexSegment, MultiIndexSegment or StreamingSegment (no index). In > future, developer is free to create MultiIndexSegment to select index > internally. Is this option better? > > Regards, > Jacky > > > 在 2016年10月3日，上午11:00，Jacky Li <[hidden email]> 写道： > > > > I am currently thinking these abstractions: > > > > - A SegmentManager is the global manager of all segments for one table. > It can be used to get all segments and manage the segment while loading and > compaction. > > - A CarbonInputFormat will take the input of table path, so means it > represent the whole table contain all segments. When getSplit is called, > it will get all segments by calling SegmentManager interface. > > - Each Segment contains a list of Index, and an IndexSelector. While > currently CarbonData only has MDK index, developer can create multiple > indices for each segment in the future. > > - An Index is an interface to filtering on block/blocklet, and provide > this functionality only. Implementation should hide all complexity like > deciding where to store the index. > > - An IndexSelector is an interface to choose which index to use based on > query predicates. Default implementation is to choose the first index. An > implementation of IndexChooser can also decide not to use index at all. > > - A Distributor is used to map the filtered block/blocklet to > InputSplits. Implementation can take number of node, parallelism into > consideration. It can also decide to distribute tasks based on block or > blocklet. > > > > So the main concepts are SegmentManager, Segment, Index, IndexSelector, > InputFormat/OutputFormat, Distributor. > > > > There will be a default implementation of CarbonInputFormat whose > getSplit will do the following: > > 1. gat all segments by calling SegmentManager > > 2. for each segment, choose the index to use by IndexSelector > > 3. invoke the selected Index to filter out block/blocklet (since these > are two concept, maybe a parent class need to be created to encapsulate > them) > > 4. distribute the filtered block/blocklet to InputSplits by Distributor. > > > > Regarding the input to the Index.filter interface, I have not decided to > use the existing QueryModel or create a new cleaner QueryModel interface. > If new QueryModel is desired, it should only contain filter predicate and > project columns, so it is much simpler than current QueryModel. But I see > current QueryModel is used in Compaction also, so I think it is better to > do this clean up later? > > > > > > Does this look fine to you? Any suggestion is welcome. > > > > Regards, > > Jacky > > > > > >> 在 2016年10月3日，上午2:18，Venkata Gollamudi <[hidden email]> 写道： > >> > >> Yes Jacky, interfaces needs to be revisited. > >> For Goal 1 and Goal 2: abstraction required for both Index and Index > store. > >> Also multi-column index(composite index) needs to be considered. > >> > >> Regards, > >> Ramana > >> > >> On Sat, Oct 1, 2016 at 11:01 AM, Jacky Li <[hidden email]> wrote: > >> > >>> Hi community, > >>> > >>> Currently CarbonData have builtin index support which is one of the > key > >>> strength of CarbonData. Using index, CarbonData can do very fast filter > >>> query by filtering on block and blocklet level. However, it also > introduces > >>> memory consumption of the index tree and impact first query time > because > >>> the > >>> process of loading of index from file footer into memory. On the other > >>> side, > >>> in a multi-tennant environment, multiple applications may access data > files > >>> simultaneously, which again exacerbate this resource consumption issue. > >>> So, I want to propose and discuss a solution with you to solve this > >>> problem and make an abstraction of interface for CarbonData's future > >>> evolvement. > >>> I am thinking the final result of this work should achieve at least > two > >>> goals: > >>> > >>> Goal 1: User can choose the place to store Index data, it can be > stored in > >>> processing framework's memory space (like in spark driver memory) or in > >>> another service outside of the processing framework (like using a > >>> independent database service) > >>> > >>> Goal 2: Developer can add more index of his choice to CarbonData files. > >>> Besides B+ tree on multi-dimensional key which current CarbonData > supports, > >>> developers are free to add other indexing technology to make certain > >>> workload faster. These new indices should be added in a pluggable way. > >>> > >>> In order to achieve these goals, an abstraction need to be created > for > >>> CarbonData project, including: > >>> > >>> - Segment: each segment is presenting one load of data, and tie with > some > >>> indices created with this load > >>> > >>> - Index: index is created when this segment is created, and is > leveraged > >>> when CarbonInputFormat's getSplit is called, to filter out the required > >>> blocks or even blocklets. > >>> > >>> - CarbonInputFormat: There maybe n number of indices created for data > file, > >>> when querying these data files, InputFormat should know how to access > these > >>> indices, and initialize or loading these index if required. > >>> > >>> Obviously, this work should be separated into different tasks and > >>> implemented gradually. But first of all, let's discuss on the goal and > the > >>> proposed approach. What is your idea? > >>> > >>> Regards, > >>> Jacky > >>> > >>> > >>> > >>> > >>> > >>> -- > >>> View this message in context: http://apache-carbondata- > >>> mailing-list-archive.1130556.n5.nabble.com/Abstracting- > >>> CarbonData-s-Index-Interface-tp1587.html > >>> Sent from the Apache CarbonData Mailing List archive mailing list > archive > >>> at Nabble.com. > >>> > > > > |
|
|
Agreed. Shall I create a JIRA issue and PR for this abstraction? I think reviewing on the interface code will be clearer. Jacky
|
Re: Abstracting CarbonData's Index Interface
|
|
Hi Jacky,
I am also changing the carbondata file thrift structure to read less and only required data while loading the btree, Main changes will be removing the data chunk from blocklet info and keeping only the offset of the data chunk and some of the redundant information which is already present in the carbonindex file i am removing from carbon data file like segment info. Only required and valid data chunk will be read during scanning. -Regards Kumar Vishal On Mon, Oct 3, 2016 at 1:08 PM, Jacky Li <[hidden email]> wrote: > Agreed. Shall I create a JIRA issue and PR for this abstraction? > I think reviewing on the interface code will be clearer. > > Regards, > Jacky > > > 在 2016年10月3日，下午2:38，Aniket Adnaik [via Apache CarbonData Mailing List > archive] <[hidden email]> 写道： > > > > I would agree with having simple segment definition. Segment can use a > > metadata info that describes the segment - For example; Segment type, > index > > availability, index type, index storage type (attached or > > detached/secondary) etc. For streaming ingest segment, it also may > possibly > > contain min-max kind of information for each blocklet, that can used for > > indexing. > > So implementation details of different segment types can be hidden from > > user. > > We may have to think about partitioning support along with load segments > in > > future. > > > > Best Regards, > > Aniket > > > > > > > > On Sun, Oct 2, 2016 at 10:25 PM, Jacky Li <[hidden email] > <x-msg://8/user/SendEmail.jtp?type=node&node=1598&i=0>> wrote: > > > > > After a second thought regarding the index part, another option is > that to > > > have a very simple Segment definition which can only list all files it > has > > > or listFile taking the QueryModel as input, implementation of Segment > can > > > be IndexSegment, MultiIndexSegment or StreamingSegment (no index). In > > > future, developer is free to create MultiIndexSegment to select index > > > internally. Is this option better? > > > > > > Regards, > > > Jacky > > > > > > > 在 2016年10月3日，上午11:00，Jacky Li <[hidden email] > <x-msg://8/user/SendEmail.jtp?type=node&node=1598&i=1>> 写道： > > > > > > > > I am currently thinking these abstractions: > > > > > > > > - A SegmentManager is the global manager of all segments for one > table. > > > It can be used to get all segments and manage the segment while > loading and > > > compaction. > > > > - A CarbonInputFormat will take the input of table path, so means it > > > represent the whole table contain all segments. When getSplit is > called, > > > it will get all segments by calling SegmentManager interface. > > > > - Each Segment contains a list of Index, and an IndexSelector. While > > > currently CarbonData only has MDK index, developer can create multiple > > > indices for each segment in the future. > > > > - An Index is an interface to filtering on block/blocklet, and > provide > > > this functionality only. Implementation should hide all complexity > like > > > deciding where to store the index. > > > > - An IndexSelector is an interface to choose which index to use > based on > > > query predicates. Default implementation is to choose the first index. > An > > > implementation of IndexChooser can also decide not to use index at all. > > > > - A Distributor is used to map the filtered block/blocklet to > > > InputSplits. Implementation can take number of node, parallelism into > > > consideration. It can also decide to distribute tasks based on block or > > > blocklet. > > > > > > > > So the main concepts are SegmentManager, Segment, Index, > IndexSelector, > > > InputFormat/OutputFormat, Distributor. > > > > > > > > There will be a default implementation of CarbonInputFormat whose > > > getSplit will do the following: > > > > 1. gat all segments by calling SegmentManager > > > > 2. for each segment, choose the index to use by IndexSelector > > > > 3. invoke the selected Index to filter out block/blocklet (since > these > > > are two concept, maybe a parent class need to be created to encapsulate > > > them) > > > > 4. distribute the filtered block/blocklet to InputSplits by > Distributor. > > > > > > > > Regarding the input to the Index.filter interface, I have not > decided to > > > use the existing QueryModel or create a new cleaner QueryModel > interface. > > > If new QueryModel is desired, it should only contain filter predicate > and > > > project columns, so it is much simpler than current QueryModel. But I > see > > > current QueryModel is used in Compaction also, so I think it is better > to > > > do this clean up later? > > > > > > > > > > > > Does this look fine to you? Any suggestion is welcome. > > > > > > > > Regards, > > > > Jacky > > > > > > > > > > > >> 在 2016年10月3日，上午2:18，Venkata Gollamudi <[hidden email] > <x-msg://8/user/SendEmail.jtp?type=node&node=1598&i=2>> 写道： > > > >> > > > >> Yes Jacky, interfaces needs to be revisited. > > > >> For Goal 1 and Goal 2: abstraction required for both Index and Index > > > store. > > > >> Also multi-column index(composite index) needs to be considered. > > > >> > > > >> Regards, > > > >> Ramana > > > >> > > > >> On Sat, Oct 1, 2016 at 11:01 AM, Jacky Li <[hidden email] > <x-msg://8/user/SendEmail.jtp?type=node&node=1598&i=3>> wrote: > > > >> > > > >>> Hi community, > > > >>> > > > >>> Currently CarbonData have builtin index support which is one of > the > > > key > > > >>> strength of CarbonData. Using index, CarbonData can do very fast > filter > > > >>> query by filtering on block and blocklet level. However, it also > > > introduces > > > >>> memory consumption of the index tree and impact first query time > > > because > > > >>> the > > > >>> process of loading of index from file footer into memory. On the > other > > > >>> side, > > > >>> in a multi-tennant environment, multiple applications may access > data > > > files > > > >>> simultaneously, which again exacerbate this resource consumption > issue. > > > >>> So, I want to propose and discuss a solution with you to solve > this > > > >>> problem and make an abstraction of interface for CarbonData's > future > > > >>> evolvement. > > > >>> I am thinking the final result of this work should achieve at > least > > > two > > > >>> goals: > > > >>> > > > >>> Goal 1: User can choose the place to store Index data, it can be > > > stored in > > > >>> processing framework's memory space (like in spark driver memory) > or in > > > >>> another service outside of the processing framework (like using a > > > >>> independent database service) > > > >>> > > > >>> Goal 2: Developer can add more index of his choice to CarbonData > files. > > > >>> Besides B+ tree on multi-dimensional key which current CarbonData > > > supports, > > > >>> developers are free to add other indexing technology to make > certain > > > >>> workload faster. These new indices should be added in a pluggable > way. > > > >>> > > > >>> In order to achieve these goals, an abstraction need to be > created > > > for > > > >>> CarbonData project, including: > > > >>> > > > >>> - Segment: each segment is presenting one load of data, and tie > with > > > some > > > >>> indices created with this load > > > >>> > > > >>> - Index: index is created when this segment is created, and is > > > leveraged > > > >>> when CarbonInputFormat's getSplit is called, to filter out the > required > > > >>> blocks or even blocklets. > > > >>> > > > >>> - CarbonInputFormat: There maybe n number of indices created for > data > > > file, > > > >>> when querying these data files, InputFormat should know how to > access > > > these > > > >>> indices, and initialize or loading these index if required. > > > >>> > > > >>> Obviously, this work should be separated into different tasks and > > > >>> implemented gradually. But first of all, let's discuss on the goal > and > > > the > > > >>> proposed approach. What is your idea? > > > >>> > > > >>> Regards, > > > >>> Jacky > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> -- > > > >>> View this message in context: http://apache-carbondata- < > http://apache-carbondata-/> > > > >>> mailing-list-archive.1130556.n5.nabble.com/Abstracting- < > http://mailing-list-archive.1130556.n5.nabble.com/Abstracting-> > > > >>> CarbonData-s-Index-Interface-tp1587.html > > > >>> Sent from the Apache CarbonData Mailing List archive mailing list > > > archive > > > >>> at Nabble.com <http://nabble.com/>. > > > >>> > > > > > > > > > > > > > > > > If you reply to this email, your message will be added to the discussion > below: > > http://apache-carbondata-mailing-list-archive.1130556. > n5.nabble.com/Abstracting-CarbonData-s-Index-Interface-tp1587p1598.html < > http://apache-carbondata-mailing-list-archive.1130556. > n5.nabble.com/Abstracting-CarbonData-s-Index-Interface-tp1587p1598.html> > > To unsubscribe from Abstracting CarbonData's Index Interface, click here > <http://apache-carbondata-mailing-list-archive.1130556. > n5.nabble.com/template/NamlServlet.jtp?macro= > unsubscribe_by_code&node=1587&code=amFja3kubGlrdW5AcXEuY29tfDE1OD > d8LTEyNTA5Nzc4Mjg=>. > > NAML <http://apache-carbondata-mailing-list-archive.1130556. > n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html% > 21nabble%3Aemail.naml&base=nabble.naml.namespaces. > BasicNamespace-nabble.view.web.template.NabbleNamespace- > nabble.naml.namespaces.BasicNamespace-nabble.view. > web.template.NabbleNamespace-nabble.naml.namespaces. > BasicNamespace-nabble.view.web.template.NabbleNamespace- > nabble.naml.namespaces.BasicNamespace-nabble.view. > web.template.NabbleNamespace-nabble.naml.namespaces. > BasicNamespace-nabble.view.web.template.NabbleNamespace- > nabble.view.web.template.NodeNamespace&breadcrumbs= > notify_subscribers%21nabble%3Aemail.naml-instant_emails% > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > > > -- > View this message in context: http://apache-carbondata- > mailing-list-archive.1130556.n5.nabble.com/Abstracting- > CarbonData-s-Index-Interface-tp1587p1599.html > Sent from the Apache CarbonData Mailing List archive mailing list archive > at Nabble.com. >
kumar vishal
|
|
|
Sure, I think what I am doing will not affect how index is stored and load for the current in memory B tree approach, I am only adding the interface for it. You can go ahead and continue your part. Regards, Jacky
|
|
|
I have created a JIRA and a PR for this:
CARBONDATA-284 (https://issues.apache.org/jira/browse/CARBONDATA-284) PR208 (https://github.com/apache/incubator-carbondata/pull/208) Please review the interface Regards, Jacky |
Re: Abstracting CarbonData's Index Interface
|
|
In reply to this post by Jacky Li
On Fri, Sep 30, 2016 at 10:31 PM, Jacky Li <[hidden email]> wrote:
> However, it also introduces memory consumption of the index tree and > impact first query time because the process of loading of index from > file footer into memory. On the other side, in a multi-tennant > environment, multiple applications may access data files simultaneously, > which again exacerbate this resource consumption issue. > Agree we shall at least not rely so much on driver memory for indexing. > > Goal 1: User can choose the place to store Index data, it can be stored > in processing framework's memory space (like in spark driver memory) or > in another service outside of the processing framework (like using a > independent database service) > How much will be the same index on different "places" code shared? For example, for Btree index, if you do it inside Carbon, you are programming at block level and you will worry about block [de]allocation, tree balance etc. But if you rely on a database service, you programming at table level, which you are programming with relational table/index. Meanwhile, index is essentially a data redundancy, so updates needs careful design if the index is outside of your control. Regards, Qingqing |
|
|
In reply to this post by Jacky Li
It is a great idea to open the door for more flexible/scalable way of accessing index to help with query processing. if our goals are as following:
> Goal 1: User can choose the place to store Index data, it can be stored in processing framework's memory space (like in spark driver memory) or in another service outside of the processing framework (like using a independent database service) the current approach of leveraging index for filter predicate/column projection pushdown through index is invisible to optimizer, since index can live outside of the current processing framework and physically separated from the base data, I am inclining with exposing index scan visible to optimizer, and allow optimizer to weight in and decide which index to pick, considering we would also like to extend the support beyond b+ tree as described in Goal 2). > Goal 2: Developer can add more index of his choice to CarbonData files. Besides B+ tree on multi-dimensional key which current CarbonData supports, developers are free to add other indexing technology to make certain workload faster. These new indices should be added in a pluggable way. Great to make it easily extensible, we can define a clean abstract API to unify the interface (build/insert/delete/scan..), refer to PostgreSQL on this matter(thanks Qingqing for sharing the link): https://www.postgresql.org/docs/8.4/static/indexam.html. Jenny -----Original Message----- From: Jacky Li [mailto:[hidden email]] Sent: Sunday, October 02, 2016 10:25 PM To: [hidden email] Subject: Re: Abstracting CarbonData's Index Interface After a second thought regarding the index part, another option is that to have a very simple Segment definition which can only list all files it has or listFile taking the QueryModel as input, implementation of Segment can be IndexSegment, MultiIndexSegment or StreamingSegment (no index). In future, developer is free to create MultiIndexSegment to select index internally. Is this option better? Regards, Jacky > 在 2016年10月3日，上午11:00，Jacky Li <[hidden email]> 写道： > > I am currently thinking these abstractions: > > - A SegmentManager is the global manager of all segments for one table. It can be used to get all segments and manage the segment while loading and compaction. > - A CarbonInputFormat will take the input of table path, so means it represent the whole table contain all segments. When getSplit is called, it will get all segments by calling SegmentManager interface. > - Each Segment contains a list of Index, and an IndexSelector. While currently CarbonData only has MDK index, developer can create multiple indices for each segment in the future. > - An Index is an interface to filtering on block/blocklet, and provide this functionality only. Implementation should hide all complexity like deciding where to store the index. > - An IndexSelector is an interface to choose which index to use based on query predicates. Default implementation is to choose the first index. An implementation of IndexChooser can also decide not to use index at all. > - A Distributor is used to map the filtered block/blocklet to InputSplits. Implementation can take number of node, parallelism into consideration. It can also decide to distribute tasks based on block or blocklet. > > So the main concepts are SegmentManager, Segment, Index, IndexSelector, InputFormat/OutputFormat, Distributor. > > There will be a default implementation of CarbonInputFormat whose getSplit will do the following: > 1. gat all segments by calling SegmentManager > 2. for each segment, choose the index to use by IndexSelector > 3. invoke the selected Index to filter out block/blocklet (since these are two concept, maybe a parent class need to be created to encapsulate them) > 4. distribute the filtered block/blocklet to InputSplits by Distributor. > > Regarding the input to the Index.filter interface, I have not decided to use the existing QueryModel or create a new cleaner QueryModel interface. If new QueryModel is desired, it should only contain filter predicate and project columns, so it is much simpler than current QueryModel. But I see current QueryModel is used in Compaction also, so I think it is better to do this clean up later? > > > Does this look fine to you? Any suggestion is welcome. > > Regards, > Jacky > > >> 在 2016年10月3日，上午2:18，Venkata Gollamudi <[hidden email]> 写道： >> >> Yes Jacky, interfaces needs to be revisited. >> For Goal 1 and Goal 2: abstraction required for both Index and Index store. >> Also multi-column index(composite index) needs to be considered. >> >> Regards, >> Ramana >> >> On Sat, Oct 1, 2016 at 11:01 AM, Jacky Li <[hidden email]> wrote: >> >>> Hi community, >>> >>> Currently CarbonData have builtin index support which is one of the key >>> strength of CarbonData. Using index, CarbonData can do very fast filter >>> query by filtering on block and blocklet level. However, it also introduces >>> memory consumption of the index tree and impact first query time because >>> the >>> process of loading of index from file footer into memory. On the other >>> side, >>> in a multi-tennant environment, multiple applications may access data files >>> simultaneously, which again exacerbate this resource consumption issue. >>> So, I want to propose and discuss a solution with you to solve this >>> problem and make an abstraction of interface for CarbonData's future >>> evolvement. >>> I am thinking the final result of this work should achieve at least two >>> goals: >>> >>> Goal 1: User can choose the place to store Index data, it can be stored in >>> processing framework's memory space (like in spark driver memory) or in >>> another service outside of the processing framework (like using a >>> independent database service) >>> >>> Goal 2: Developer can add more index of his choice to CarbonData files. >>> Besides B+ tree on multi-dimensional key which current CarbonData supports, >>> developers are free to add other indexing technology to make certain >>> workload faster. These new indices should be added in a pluggable way. >>> >>> In order to achieve these goals, an abstraction need to be created for >>> CarbonData project, including: >>> >>> - Segment: each segment is presenting one load of data, and tie with some >>> indices created with this load >>> >>> - Index: index is created when this segment is created, and is leveraged >>> when CarbonInputFormat's getSplit is called, to filter out the required >>> blocks or even blocklets. >>> >>> - CarbonInputFormat: There maybe n number of indices created for data file, >>> when querying these data files, InputFormat should know how to access these >>> indices, and initialize or loading these index if required. >>> >>> Obviously, this work should be separated into different tasks and >>> implemented gradually. But first of all, let's discuss on the goal and the >>> proposed approach. What is your idea? >>> >>> Regards, >>> Jacky >>> >>> >>> >>> >>> >>> -- >>> View this message in context: http://apache-carbondata- >>> mailing-list-archive.1130556.n5.nabble.com/Abstracting- >>> CarbonData-s-Index-Interface-tp1587.html >>> Sent from the Apache CarbonData Mailing List archive mailing list archive >>> at Nabble.com. >>> > |
|
|
In reply to this post by Qingqing Zhou
> 在 2016年10月4日，上午5:43，Qingqing Zhou <[hidden email]> 写道： > > On Fri, Sep 30, 2016 at 10:31 PM, Jacky Li <[hidden email]> wrote: >> However, it also introduces memory consumption of the index tree and >> impact first query time because the process of loading of index from >> file footer into memory. On the other side, in a multi-tennant >> environment, multiple applications may access data files simultaneously, >> which again exacerbate this resource consumption issue. >> > Agree we shall at least not rely so much on driver memory for indexing. > >> >> Goal 1: User can choose the place to store Index data, it can be stored >> in processing framework's memory space (like in spark driver memory) or >> in another service outside of the processing framework (like using a >> independent database service) >> > > How much will be the same index on different "places" code shared? For > example, for Btree index, if you do it inside Carbon, you are programming > at block level and you will worry about block [de]allocation, tree balance > etc. But if you rely on a database service, you programming at table > level, which you are programming with relational table/index. Meanwhile, > index is essentially a data redundancy, so updates needs careful design if > the index is outside of your control. > I think we can try to reuse anything except for Index storage, like segment management, query logic processing after InputSplit is gathered by calling index interface. I think index can be programmed in different level, what I proposed here is still a block level solution, so it can be processed in InputFormat level. If you are looking for table level indexing solution, it means that you need to manipulate the query plan to do some kind of join of two tables, so means we need to add logic into processing framework’s optimizer which I tend to avoid in CarbonData project, unless it has huge benefits. Because every optimizer is having different interface, there are no *standard* way to do it right now. Do you see any benefit of doing it in table level? > Regards, > Qingqing > |
|
|
In reply to this post by Jihong Ma
> 在 2016年10月4日，上午8:01，Jihong Ma <[hidden email]> 写道： > > It is a great idea to open the door for more flexible/scalable way of accessing index to help with query processing. if our goals are as following: > >> Goal 1: User can choose the place to store Index data, it can be stored in processing framework's memory space (like in spark driver memory) or in > another service outside of the processing framework (like using a independent database service) > > the current approach of leveraging index for filter predicate/column projection pushdown through index is invisible to optimizer, since index can live outside of the current processing framework and physically separated from the base data, I am inclining with exposing index scan visible to optimizer, and allow optimizer to weight in and decide which index to pick, considering we would also like to extend the support beyond b+ tree as described in Goal 2). > I think the bottom line is that CarbonData index should be easy enough to integrate into different processing frameworks, which I think InputFormat is the best choice as of now. Integration in this level, enable all processing framework can leverage CarbonData’s index capability without any plugin code in the upper framework. The next level of integration is in optimizer level, in this case, unless the optimizer is capable of evaluating cost of using different indices, exposing the index as API to optimizer is meaningful. I think we can always to do that later on, if we have added more indices to CarbonData and come up with a neat API for optimizer to use. > >> Goal 2: Developer can add more index of his choice to CarbonData files. > Besides B+ tree on multi-dimensional key which current CarbonData supports, developers are free to add other indexing technology to make certain workload faster. These new indices should be added in a pluggable way. > > Great to make it easily extensible, we can define a clean abstract API to unify the interface (build/insert/delete/scan..), refer to PostgreSQL on this matter(thanks Qingqing for sharing the link): > https://www.postgresql.org/docs/8.4/static/indexam.html. > I think you are right, we should have a clean Index API, currently I am thinking of read and load API only: https://github.com/apache/incubator-carbondata/pull/208/files#diff-ec23f9a4b91d70309522eef78689be1c <https://github.com/apache/incubator-carbondata/pull/208/files#diff-ec23f9a4b91d70309522eef78689be1c> For the traditional database like PostgreSQL, since they need to deal with transactions so their index API need to consider transaction as well, while in CarbonData, I am not sure how far we can go and how much mutability we can add to Index API. Currently, its model is using Segment concept to do incremental load, so Index is immutable. If data is append only, I think read and load of index is enough. But if we introduce update into CarbonData, then index should be updatable as well, I think it is better to consider this together in data update feature if there are any in the future? > > Jenny > > -----Original Message----- > From: Jacky Li [mailto:[hidden email]] > Sent: Sunday, October 02, 2016 10:25 PM > To: [hidden email] > Subject: Re: Abstracting CarbonData's Index Interface > > After a second thought regarding the index part, another option is that to have a very simple Segment definition which can only list all files it has or listFile taking the QueryModel as input, implementation of Segment can be IndexSegment, MultiIndexSegment or StreamingSegment (no index). In future, developer is free to create MultiIndexSegment to select index internally. Is this option better? > > Regards, > Jacky > >> 在 2016年10月3日，上午11:00，Jacky Li <[hidden email]> 写道： >> >> I am currently thinking these abstractions: >> >> - A SegmentManager is the global manager of all segments for one table. It can be used to get all segments and manage the segment while loading and compaction. >> - A CarbonInputFormat will take the input of table path, so means it represent the whole table contain all segments. When getSplit is called, it will get all segments by calling SegmentManager interface. >> - Each Segment contains a list of Index, and an IndexSelector. While currently CarbonData only has MDK index, developer can create multiple indices for each segment in the future. >> - An Index is an interface to filtering on block/blocklet, and provide this functionality only. Implementation should hide all complexity like deciding where to store the index. >> - An IndexSelector is an interface to choose which index to use based on query predicates. Default implementation is to choose the first index. An implementation of IndexChooser can also decide not to use index at all. >> - A Distributor is used to map the filtered block/blocklet to InputSplits. Implementation can take number of node, parallelism into consideration. It can also decide to distribute tasks based on block or blocklet. >> >> So the main concepts are SegmentManager, Segment, Index, IndexSelector, InputFormat/OutputFormat, Distributor. >> >> There will be a default implementation of CarbonInputFormat whose getSplit will do the following: >> 1. gat all segments by calling SegmentManager >> 2. for each segment, choose the index to use by IndexSelector >> 3. invoke the selected Index to filter out block/blocklet (since these are two concept, maybe a parent class need to be created to encapsulate them) >> 4. distribute the filtered block/blocklet to InputSplits by Distributor. >> >> Regarding the input to the Index.filter interface, I have not decided to use the existing QueryModel or create a new cleaner QueryModel interface. If new QueryModel is desired, it should only contain filter predicate and project columns, so it is much simpler than current QueryModel. But I see current QueryModel is used in Compaction also, so I think it is better to do this clean up later? >> >> >> Does this look fine to you? Any suggestion is welcome. >> >> Regards, >> Jacky >> >> >>> 在 2016年10月3日，上午2:18，Venkata Gollamudi <[hidden email]> 写道： >>> >>> Yes Jacky, interfaces needs to be revisited. >>> For Goal 1 and Goal 2: abstraction required for both Index and Index store. >>> Also multi-column index(composite index) needs to be considered. >>> >>> Regards, >>> Ramana >>> >>> On Sat, Oct 1, 2016 at 11:01 AM, Jacky Li <[hidden email]> wrote: >>> >>>> Hi community, >>>> >>>> Currently CarbonData have builtin index support which is one of the key >>>> strength of CarbonData. Using index, CarbonData can do very fast filter >>>> query by filtering on block and blocklet level. However, it also introduces >>>> memory consumption of the index tree and impact first query time because >>>> the >>>> process of loading of index from file footer into memory. On the other >>>> side, >>>> in a multi-tennant environment, multiple applications may access data files >>>> simultaneously, which again exacerbate this resource consumption issue. >>>> So, I want to propose and discuss a solution with you to solve this >>>> problem and make an abstraction of interface for CarbonData's future >>>> evolvement. >>>> I am thinking the final result of this work should achieve at least two >>>> goals: >>>> >>>> Goal 1: User can choose the place to store Index data, it can be stored in >>>> processing framework's memory space (like in spark driver memory) or in >>>> another service outside of the processing framework (like using a >>>> independent database service) >>>> >>>> Goal 2: Developer can add more index of his choice to CarbonData files. >>>> Besides B+ tree on multi-dimensional key which current CarbonData supports, >>>> developers are free to add other indexing technology to make certain >>>> workload faster. These new indices should be added in a pluggable way. >>>> >>>> In order to achieve these goals, an abstraction need to be created for >>>> CarbonData project, including: >>>> >>>> - Segment: each segment is presenting one load of data, and tie with some >>>> indices created with this load >>>> >>>> - Index: index is created when this segment is created, and is leveraged >>>> when CarbonInputFormat's getSplit is called, to filter out the required >>>> blocks or even blocklets. >>>> >>>> - CarbonInputFormat: There maybe n number of indices created for data file, >>>> when querying these data files, InputFormat should know how to access these >>>> indices, and initialize or loading these index if required. >>>> >>>> Obviously, this work should be separated into different tasks and >>>> implemented gradually. But first of all, let's discuss on the goal and the >>>> proposed approach. What is your idea? >>>> >>>> Regards, >>>> Jacky >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: http://apache-carbondata- >>>> mailing-list-archive.1130556.n5.nabble.com/Abstracting- >>>> CarbonData-s-Index-Interface-tp1587.html >>>> Sent from the Apache CarbonData Mailing List archive mailing list archive >>>> at Nabble.com. >>>> >> > |
Re: Abstracting CarbonData's Index Interface
|
|
In reply to this post by Jacky Li
On Mon, Oct 3, 2016 at 8:52 PM, Jacky Li <[hidden email]> wrote:
> I think we can try to reuse anything except for Index storage, like > segment management, query logic processing after InputSplit is gathered > by calling index interface.I think index can be programmed in different > level, what I proposed here is still a block level solution, so it can > be processed in InputFormat level. I agree the scan provider is at InputFormat level, this is the same for wherever you store the index. What the discrepancy I see here is the implementation of the index itself: if you use "database service" to store your index, you can simple invoke an "CRAETE INDEX" statement to implment indexing, but if you want to store index in Carbon, you will need to implment the btree yourself. Agree "segment management" can be shared, as it is for the indexed data. About "query logic processing": Currently Carbon pushes certain SARGs into storage level: so in above picture, these two implementations won't be able to share this logic: the "data service" one will rely on the query processor (currently Spark) to tell how to use index, while "carbon" one will handle it internally. To change this, we will have to expose "carbon" index to query processor level. Regards, Qingqing |
Re: Abstracting CarbonData's Index Interface
|
|
In reply to this post by Jacky Li
On Mon, Oct 3, 2016 at 9:14 PM, Jacky Li <[hidden email]> wrote:
> If data is append only, I think read and load of index is enough. I > think you are right, we should have a clean Index API, currently I am > thinking of read and load API only: I am not super fan for external index at this stage. Comparing to "database" style supporting of external index involving complex/high performance transaction requirements, we can alleviate a bit. But still, even for loading, basic transaction requirement like atomic is still needed. For now, we can focus more on internal index interface, so carbon kernel developers can introduce new index easier. Regards, Qingqing |
|
|
In reply to this post by Jacky Li
we are all on the same page w.r.t inputFormat interface to perform predicate /column projection pushdown with help of index at FILE LEVEL, for achieving Goal 2) specifically. For achieving Goal 1), to realize query predicate/column pruning at TABLE LEVEL through leveraging index (of any form, no matter where it is stored), this should be achieved at query planning phase, and exposed to optimizer to pick the right plan based on cost(whether to leverage index, pick which index?). Jenny -----Original Message----- From: Jacky Li [mailto:[hidden email]] Sent: Monday, October 03, 2016 9:15 PM To: [hidden email] Subject: Re: Abstracting CarbonData's Index Interface > 在 2016年10月4日，上午8:01，Jihong Ma <[hidden email]> 写道： > > It is a great idea to open the door for more flexible/scalable way of accessing index to help with query processing. if our goals are as following: > >> Goal 1: User can choose the place to store Index data, it can be stored in processing framework's memory space (like in spark driver memory) or in > another service outside of the processing framework (like using a independent database service) > > the current approach of leveraging index for filter predicate/column projection pushdown through index is invisible to optimizer, since index can live outside of the current processing framework and physically separated from the base data, I am inclining with exposing index scan visible to optimizer, and allow optimizer to weight in and decide which index to pick, considering we would also like to extend the support beyond b+ tree as described in Goal 2). > I think the bottom line is that CarbonData index should be easy enough to integrate into different processing frameworks, which I think InputFormat is the best choice as of now. Integration in this level, enable all processing framework can leverage CarbonData’s index capability without any plugin code in the upper framework. The next level of integration is in optimizer level, in this case, unless the optimizer is capable of evaluating cost of using different indices, exposing the index as API to optimizer is meaningful. I think we can always to do that later on, if we have added more indices to CarbonData and come up with a neat API for optimizer to use. > >> Goal 2: Developer can add more index of his choice to CarbonData files. > Besides B+ tree on multi-dimensional key which current CarbonData supports, developers are free to add other indexing technology to make certain workload faster. These new indices should be added in a pluggable way. > > Great to make it easily extensible, we can define a clean abstract API to unify the interface (build/insert/delete/scan..), refer to PostgreSQL on this matter(thanks Qingqing for sharing the link): > https://www.postgresql.org/docs/8.4/static/indexam.html. > I think you are right, we should have a clean Index API, currently I am thinking of read and load API only: https://github.com/apache/incubator-carbondata/pull/208/files#diff-ec23f9a4b91d70309522eef78689be1c <https://github.com/apache/incubator-carbondata/pull/208/files#diff-ec23f9a4b91d70309522eef78689be1c> For the traditional database like PostgreSQL, since they need to deal with transactions so their index API need to consider transaction as well, while in CarbonData, I am not sure how far we can go and how much mutability we can add to Index API. Currently, its model is using Segment concept to do incremental load, so Index is immutable. If data is append only, I think read and load of index is enough. But if we introduce update into CarbonData, then index should be updatable as well, I think it is better to consider this together in data update feature if there are any in the future? > > Jenny > > -----Original Message----- > From: Jacky Li [mailto:[hidden email]] > Sent: Sunday, October 02, 2016 10:25 PM > To: [hidden email] > Subject: Re: Abstracting CarbonData's Index Interface > > After a second thought regarding the index part, another option is that to have a very simple Segment definition which can only list all files it has or listFile taking the QueryModel as input, implementation of Segment can be IndexSegment, MultiIndexSegment or StreamingSegment (no index). In future, developer is free to create MultiIndexSegment to select index internally. Is this option better? > > Regards, > Jacky > >> 在 2016年10月3日，上午11:00，Jacky Li <[hidden email]> 写道： >> >> I am currently thinking these abstractions: >> >> - A SegmentManager is the global manager of all segments for one table. It can be used to get all segments and manage the segment while loading and compaction. >> - A CarbonInputFormat will take the input of table path, so means it represent the whole table contain all segments. When getSplit is called, it will get all segments by calling SegmentManager interface. >> - Each Segment contains a list of Index, and an IndexSelector. While currently CarbonData only has MDK index, developer can create multiple indices for each segment in the future. >> - An Index is an interface to filtering on block/blocklet, and provide this functionality only. Implementation should hide all complexity like deciding where to store the index. >> - An IndexSelector is an interface to choose which index to use based on query predicates. Default implementation is to choose the first index. An implementation of IndexChooser can also decide not to use index at all. >> - A Distributor is used to map the filtered block/blocklet to InputSplits. Implementation can take number of node, parallelism into consideration. It can also decide to distribute tasks based on block or blocklet. >> >> So the main concepts are SegmentManager, Segment, Index, IndexSelector, InputFormat/OutputFormat, Distributor. >> >> There will be a default implementation of CarbonInputFormat whose getSplit will do the following: >> 1. gat all segments by calling SegmentManager >> 2. for each segment, choose the index to use by IndexSelector >> 3. invoke the selected Index to filter out block/blocklet (since these are two concept, maybe a parent class need to be created to encapsulate them) >> 4. distribute the filtered block/blocklet to InputSplits by Distributor. >> >> Regarding the input to the Index.filter interface, I have not decided to use the existing QueryModel or create a new cleaner QueryModel interface. If new QueryModel is desired, it should only contain filter predicate and project columns, so it is much simpler than current QueryModel. But I see current QueryModel is used in Compaction also, so I think it is better to do this clean up later? >> >> >> Does this look fine to you? Any suggestion is welcome. >> >> Regards, >> Jacky >> >> >>> 在 2016年10月3日，上午2:18，Venkata Gollamudi <[hidden email]> 写道： >>> >>> Yes Jacky, interfaces needs to be revisited. >>> For Goal 1 and Goal 2: abstraction required for both Index and Index store. >>> Also multi-column index(composite index) needs to be considered. >>> >>> Regards, >>> Ramana >>> >>> On Sat, Oct 1, 2016 at 11:01 AM, Jacky Li <[hidden email]> wrote: >>> >>>> Hi community, >>>> >>>> Currently CarbonData have builtin index support which is one of the key >>>> strength of CarbonData. Using index, CarbonData can do very fast filter >>>> query by filtering on block and blocklet level. However, it also introduces >>>> memory consumption of the index tree and impact first query time because >>>> the >>>> process of loading of index from file footer into memory. On the other >>>> side, >>>> in a multi-tennant environment, multiple applications may access data files >>>> simultaneously, which again exacerbate this resource consumption issue. >>>> So, I want to propose and discuss a solution with you to solve this >>>> problem and make an abstraction of interface for CarbonData's future >>>> evolvement. >>>> I am thinking the final result of this work should achieve at least two >>>> goals: >>>> >>>> Goal 1: User can choose the place to store Index data, it can be stored in >>>> processing framework's memory space (like in spark driver memory) or in >>>> another service outside of the processing framework (like using a >>>> independent database service) >>>> >>>> Goal 2: Developer can add more index of his choice to CarbonData files. >>>> Besides B+ tree on multi-dimensional key which current CarbonData supports, >>>> developers are free to add other indexing technology to make certain >>>> workload faster. These new indices should be added in a pluggable way. >>>> >>>> In order to achieve these goals, an abstraction need to be created for >>>> CarbonData project, including: >>>> >>>> - Segment: each segment is presenting one load of data, and tie with some >>>> indices created with this load >>>> >>>> - Index: index is created when this segment is created, and is leveraged >>>> when CarbonInputFormat's getSplit is called, to filter out the required >>>> blocks or even blocklets. >>>> >>>> - CarbonInputFormat: There maybe n number of indices created for data file, >>>> when querying these data files, InputFormat should know how to access these >>>> indices, and initialize or loading these index if required. >>>> >>>> Obviously, this work should be separated into different tasks and >>>> implemented gradually. But first of all, let's discuss on the goal and the >>>> proposed approach. What is your idea? >>>> >>>> Regards, >>>> Jacky >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: http://apache-carbondata- >>>> mailing-list-archive.1130556.n5.nabble.com/Abstracting- >>>> CarbonData-s-Index-Interface-tp1587.html >>>> Sent from the Apache CarbonData Mailing List archive mailing list archive >>>> at Nabble.com. >>>> >> > |
«
Return to Apache CarbonData Dev Mailing List archive
|
1 view|%1 views
| Free forum by Nabble | Edit this page |