Re: [Discussion] Implement Lucene DataMap to support full text search

Posted by xuchuanyin on Feb 09, 2018; 8:56am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Discussion-Implement-Lucene-DataMap-to-support-full-text-search-tp38753p39155.html


Where will the code of this feature be?
I think it will lie in a separate module. It would be better to treat datamaps as plugins and not strongly involved with carbondata core/processing module.

> -----Original Messages-----
> From: "David CaiQiang" <[hidden email]>
> Sent Time: 2018-02-07 17:25:39 (Wednesday)
> To: [hidden email]
> Cc:
> Subject: [Discussion] Implement Lucene DataMap to support full text search
>
> Hi all,
>     Let's discuss to support full-text search.  
>
>     A solution is embedding Lucene search library, index text columns for
> each segment and support searching on text columns.
>
>     Listed some sub-tasks as following.
>
>     1). create Lucene DataMap with 'text_columns' property and build Lucene
> DataMap for all exists segments
>
>        create datamap <datamapName> on <tableName>
>        using 'lucene'
>        dmproperties('text_columns'='col1,col2')
>
>     2). load data should build Lucene DataMap for the segment
>
>     3). query with Lucene DataMap while filters contain match UDF
>
>     4). compaction should rebuild Lucene DataMap for the new segment
>
>     5). update and delete data should sync Lucene DataMap
>
>     6). show DataMap for Lucene DataMap
>
>     7). delete segment should remove Lucene DataMap of this segment
>
>     8). drop table should remove Lucene DataMap of all segments
>
>     9). block streaming feature if the table has Lucene DataMap
>
>     10). Pre-aggregate DataMap feature not support match UDF
>
>     Any suggestion, any question?
>
>
>
> -----
> Best Regards
> David Cai
> --
> Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/