[GitHub] carbondata pull request #1510: [WIP] Supported DataMap chooser and expressio...

classic Classic list List threaded Threaded
61 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1458/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1875/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    @jackylk Please review, I have moved the `like` expression conversion to carbon layer in this PR itself, so like expression can be accessed as `STARTSWITH` or `ENDSWITH` or `CONTAINSWITH` to the datamap


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xubochin commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    Thought this code , I understand that : STARTSWITH is "%xxx",ENDSWITH is "%xxx",CONTAINSWITH is "xxx", is it right ?  Luceen only use STARTSWITH.


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xubochin commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    except ENDSWITH , other like string can pass to Lucene ?


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    @xubochin LuceneDataMap can only include the expression type which it supports inside metadata so that remaining expressions carbon can handle it.


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1510: [CARBONDATA-1543] Supported DataMap chooser a...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1510#discussion_r156602474
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/DataMapChooser.java ---
    @@ -0,0 +1,269 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.carbondata.core.datamap;
    +
    +import java.util.ArrayList;
    +import java.util.Collections;
    +import java.util.HashSet;
    +import java.util.List;
    +import java.util.Set;
    +
    +import org.apache.carbondata.core.datamap.dev.expr.AndDataMapExprWrapper;
    +import org.apache.carbondata.core.datamap.dev.expr.DataMapExprWrapper;
    +import org.apache.carbondata.core.datamap.dev.expr.DataMapExprWrapperImpl;
    +import org.apache.carbondata.core.datamap.dev.expr.OrDataMapExprWrapper;
    +import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
    +import org.apache.carbondata.core.scan.expression.ColumnExpression;
    +import org.apache.carbondata.core.scan.expression.Expression;
    +import org.apache.carbondata.core.scan.expression.logical.AndExpression;
    +import org.apache.carbondata.core.scan.expression.logical.OrExpression;
    +import org.apache.carbondata.core.scan.filter.intf.ExpressionType;
    +import org.apache.carbondata.core.scan.filter.resolver.FilterResolverIntf;
    +import org.apache.carbondata.core.scan.filter.resolver.resolverinfo.TrueConditionalResolverImpl;
    +
    +/**
    + * This chooser does 2 jobs.
    + * 1. Based on filter expression it converts the available datamaps to datamap expression.
    + *   For example, there are 2 datamaps available on table1
    + *   Datamap1 : column1
    + *   Datamap2 : column2
    + *   Query: select * from table1 where column1 ='a' and column2 =b
    + *   For the above query, we create datamap expression as AndDataMapExpression(Datamap1, DataMap2).
    + *   So for the above query both the datamaps are included and the output of them will be
    + *   applied AND condition to improve the performance
    + *
    + * 2. It chooses the datamap out of available datamaps based on simple logic.
    + *   Like if there is filter condition on column1 then for
    + *   supposing 2 datamaps(1. column1 2. column1+column2) are supporting this column then we choose
    + *   the datamap which has fewer columns that is the first datamap.
    + */
    +public class DataMapChooser {
    +
    +  public DataMapExprWrapper choose(CarbonTable carbonTable,
    +      FilterResolverIntf resolverIntf) {
    +    if (resolverIntf != null) {
    +      Expression expression = resolverIntf.getFilterExpression();
    +      // First check for FG datamaps if any exist
    +      List<TableDataMap> allDataMapFG =
    +          DataMapStoreManager.getInstance().getAllDataMap(carbonTable, DataMapType.FG);
    +      ExpressionTuple tuple = selectDataMap(expression, allDataMapFG);
    +      if (tuple.dataMapExprWrapper == null) {
    +        // Check for CG datamap
    +        List<TableDataMap> allDataMapCG =
    +            DataMapStoreManager.getInstance().getAllDataMap(carbonTable, DataMapType.CG);
    +        tuple = selectDataMap(expression, allDataMapCG);
    +      }
    +      if (tuple.dataMapExprWrapper != null) {
    +        return tuple.dataMapExprWrapper;
    +      }
    +    }
    +    // Return the default datamap if no other datamap exists.
    +    return new DataMapExprWrapperImpl(DataMapStoreManager.getInstance()
    --- End diff --
   
    make the line wrap easier to read, like:
    ```
        return new DataMapExprWrapperImpl(
            DataMapStoreManager.getInstance().getDefaultDataMap(
                carbonTable.getAbsoluteTableIdentifier()), resolverIntf);
    ```


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1092/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2308/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2543/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1953/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3152/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3187/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2284/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3521/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3367/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3783/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2538/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    retest this please


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1510
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3786/



---
1234