[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...

classic Classic list List threaded Threaded
39 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2411
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6636/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2411
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6638/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2411
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5466/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2411
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5504/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2411
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5534/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2411
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6679/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2411#discussion_r200017553
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapWrapperSchema.java ---
    @@ -0,0 +1,87 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.datamap.dev.expr;
    +
    +import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
    +
    +/**
    + * schema for datamap wrapper
    --- End diff --
   
    Can you describe why this is needed?


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2411#discussion_r200018301
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapWrapperSchema.java ---
    @@ -0,0 +1,87 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.datamap.dev.expr;
    +
    +import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
    +
    +/**
    + * schema for datamap wrapper
    --- End diff --
   
    Because currently a DatamapWrapper contains more than one datamap, this class is to describle its schema.
    For example a AndDatamapWrapper contains BloomFilterin its left and Lucene in its right, then its schema would be AND(BloomFilter, Lucene).
   
    This DataMapWrapperSchema  is used for explain output


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2411#discussion_r200018398
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapWrapperSchema.java ---
    @@ -0,0 +1,87 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.datamap.dev.expr;
    +
    +import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
    +
    +/**
    + * schema for datamap wrapper
    + */
    +public class DataMapWrapperSchema {
    +  enum WrapperType {
    +    PRIMITIVE,
    +    AND,
    +    OR
    +  }
    +
    +  private WrapperType wrapperType;
    +  private DataMapWrapperSchema left;
    +  private DataMapWrapperSchema right;
    +  private DataMapSchema schema;
    +
    +  private DataMapWrapperSchema(WrapperType wrapperType, DataMapWrapperSchema left,
    +      DataMapWrapperSchema right) {
    +    this.wrapperType = wrapperType;
    +    this.left = left;
    +    this.right = right;
    +  }
    +
    +  private DataMapWrapperSchema(DataMapSchema schema) {
    +    this.wrapperType = WrapperType.PRIMITIVE;
    +    this.schema = schema;
    +  }
    +
    +  public static DataMapWrapperSchema fromDataMapWrapper(DataMapExprWrapper dataMapExprWrapper) {
    +    if (dataMapExprWrapper instanceof DataMapExprWrapperImpl) {
    +      return new DataMapWrapperSchema(
    +          ((DataMapExprWrapperImpl) dataMapExprWrapper).getDataMapSchema());
    +    } else if (dataMapExprWrapper instanceof AndDataMapExprWrapper) {
    +      return new DataMapWrapperSchema(WrapperType.AND,
    +          fromDataMapWrapper(dataMapExprWrapper.getLeftDataMapWrapper()),
    +          fromDataMapWrapper(dataMapExprWrapper.getRightDataMapWrapprt()));
    +    } else {
    +      return new DataMapWrapperSchema(WrapperType.OR,
    +          fromDataMapWrapper(dataMapExprWrapper.getLeftDataMapWrapper()),
    +          fromDataMapWrapper(dataMapExprWrapper.getRightDataMapWrapprt()));
    +    }
    +  }
    +
    +  public String getDataMapWrapperName() {
    +    if (WrapperType.PRIMITIVE == wrapperType) {
    +      return schema.getDataMapName();
    +    } else {
    +      return String.format("%s(%s, %s)",
    --- End diff --
   
    FYI: This is the wrapperName


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2411#discussion_r200018440
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapWrapperSchema.java ---
    @@ -0,0 +1,87 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.datamap.dev.expr;
    +
    +import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
    +
    +/**
    + * schema for datamap wrapper
    + */
    +public class DataMapWrapperSchema {
    +  enum WrapperType {
    +    PRIMITIVE,
    +    AND,
    +    OR
    +  }
    +
    +  private WrapperType wrapperType;
    +  private DataMapWrapperSchema left;
    +  private DataMapWrapperSchema right;
    +  private DataMapSchema schema;
    +
    +  private DataMapWrapperSchema(WrapperType wrapperType, DataMapWrapperSchema left,
    +      DataMapWrapperSchema right) {
    +    this.wrapperType = wrapperType;
    +    this.left = left;
    +    this.right = right;
    +  }
    +
    +  private DataMapWrapperSchema(DataMapSchema schema) {
    +    this.wrapperType = WrapperType.PRIMITIVE;
    +    this.schema = schema;
    +  }
    +
    +  public static DataMapWrapperSchema fromDataMapWrapper(DataMapExprWrapper dataMapExprWrapper) {
    +    if (dataMapExprWrapper instanceof DataMapExprWrapperImpl) {
    +      return new DataMapWrapperSchema(
    +          ((DataMapExprWrapperImpl) dataMapExprWrapper).getDataMapSchema());
    +    } else if (dataMapExprWrapper instanceof AndDataMapExprWrapper) {
    +      return new DataMapWrapperSchema(WrapperType.AND,
    +          fromDataMapWrapper(dataMapExprWrapper.getLeftDataMapWrapper()),
    +          fromDataMapWrapper(dataMapExprWrapper.getRightDataMapWrapprt()));
    +    } else {
    +      return new DataMapWrapperSchema(WrapperType.OR,
    +          fromDataMapWrapper(dataMapExprWrapper.getLeftDataMapWrapper()),
    +          fromDataMapWrapper(dataMapExprWrapper.getRightDataMapWrapprt()));
    +    }
    +  }
    +
    +  public String getDataMapWrapperName() {
    +    if (WrapperType.PRIMITIVE == wrapperType) {
    +      return schema.getDataMapName();
    +    } else {
    +      return String.format("%s(%s, %s)",
    +          wrapperType, left.getDataMapWrapperName(), right.getDataMapWrapperName());
    +    }
    +  }
    +
    +  public String getDataMapWrapperProvider() {
    +    if (WrapperType.PRIMITIVE == wrapperType) {
    +      return schema.getProviderName();
    +    } else {
    +      return String.format("%s(%s, %s)",
    --- End diff --
   
    FYI: This is the wrapperProvider


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2411#discussion_r200018803
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapWrapperSchema.java ---
    @@ -0,0 +1,87 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.datamap.dev.expr;
    +
    +import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
    +
    +/**
    + * schema for datamap wrapper
    --- End diff --
   
    Can you add these in comment.
    And if this class is only used for Explain, can you give a better name for it.


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2411#discussion_r200021222
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapWrapperSchema.java ---
    @@ -0,0 +1,87 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.datamap.dev.expr;
    +
    +import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
    +
    +/**
    + * schema for datamap wrapper
    --- End diff --
   
    OK, will change the name to 'DataMapWrapperSimpleInfo'


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/2411
 
    The output for explain looks like below:
    ```
    Table Scan on carbon_bloom
     - total blocklets: 1
     - filter: (((((name <> null and city <> null) and id <> null) and id = 1) and city = city_1) and name = n1)
     - pruned by Main DataMap
        - skipped blocklets: 0
     - pruned by CG DataMap
        - name: AND(AND(datamap11, datamap13), datamap12)
        - provider: AND(AND(bloomfilter, bloomfilter), bloomfilter)
        - skipped blocklets: 0
    ```
    Before this PR, the output only shows `datamap11`


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2411
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5595/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2411
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6756/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2411
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5588/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/2411
 
    LGTM


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2411: [CARBONDATA-2654][Datamap] Optimize output fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/2411


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2411
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5611/



---
12