[GitHub] carbondata pull request #2148: [CARBONDATA-2323][WIP] Distributed search mod...

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2148
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3907/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2148
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5128/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2148
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3913/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2148
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5134/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2148
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5139/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2148
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3918/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2148#discussion_r182620157
 
    --- Diff: common/pom.xml ---
    @@ -52,6 +52,24 @@
         <dependency>
           <groupId>org.apache.hadoop</groupId>
           <artifactId>hadoop-common</artifactId>
    +      <exclusions>
    +        <exclusion>
    +          <groupId>io.netty</groupId>
    +          <artifactId>netty</artifactId>
    +        </exclusion>
    +        <exclusion>
    +          <groupId>io.netty</groupId>
    +          <artifactId>netty-all</artifactId>
    +        </exclusion>
    +        <exclusion>
    +          <groupId>com.google.protobuf</groupId>
    +          <artifactId>protobuf-java</artifactId>
    +        </exclusion>
    +        <exclusion>
    +          <groupId>com.google.guava</groupId>
    +          <artifactId>guava</artifactId>
    +        </exclusion>
    +      </exclusions>
    --- End diff --
   
    Since you are not using grpc these exclusions are required now?


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2148#discussion_r182620251
 
    --- Diff: dev/findbugs-exclude.xml ---
    @@ -98,4 +102,9 @@
       <Match>
         <Bug pattern="JLM_JSR166_UTILCONCURRENT_MONITORENTER"/>
       </Match>
    +
    +  <Match>
    +    <!-- gRPC generated code in search module -->
    +    <Package name="org.apache.carbondata.store.protocol"/>
    +  </Match>
    --- End diff --
   
    I think this changes are not required


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2148#discussion_r182620461
 
    --- Diff: examples/spark2/pom.xml ---
    @@ -62,6 +62,11 @@
           <version>${spark.version}</version>
           <scope>${spark.deps.scope}</scope>
         </dependency>
    +    <dependency>
    +      <groupId>com.google.guava</groupId>
    +      <artifactId>guava</artifactId>
    +      <version>19.0</version>
    +    </dependency>
    --- End diff --
   
    Is this dependency required? I don't see any special imports in example classes


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2148#discussion_r182621383
 
    --- Diff: common/pom.xml ---
    @@ -52,6 +52,24 @@
         <dependency>
           <groupId>org.apache.hadoop</groupId>
           <artifactId>hadoop-common</artifactId>
    +      <exclusions>
    +        <exclusion>
    +          <groupId>io.netty</groupId>
    +          <artifactId>netty</artifactId>
    +        </exclusion>
    +        <exclusion>
    +          <groupId>io.netty</groupId>
    +          <artifactId>netty-all</artifactId>
    +        </exclusion>
    +        <exclusion>
    +          <groupId>com.google.protobuf</groupId>
    +          <artifactId>protobuf-java</artifactId>
    +        </exclusion>
    +        <exclusion>
    +          <groupId>com.google.guava</groupId>
    +          <artifactId>guava</artifactId>
    +        </exclusion>
    +      </exclusions>
    --- End diff --
   
    fixed


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2148#discussion_r182622389
 
    --- Diff: dev/findbugs-exclude.xml ---
    @@ -98,4 +102,9 @@
       <Match>
         <Bug pattern="JLM_JSR166_UTILCONCURRENT_MONITORENTER"/>
       </Match>
    +
    +  <Match>
    +    <!-- gRPC generated code in search module -->
    +    <Package name="org.apache.carbondata.store.protocol"/>
    +  </Match>
    --- End diff --
   
    fixed


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2148#discussion_r182622408
 
    --- Diff: examples/spark2/pom.xml ---
    @@ -62,6 +62,11 @@
           <version>${spark.version}</version>
           <scope>${spark.deps.scope}</scope>
         </dependency>
    +    <dependency>
    +      <groupId>com.google.guava</groupId>
    +      <artifactId>guava</artifactId>
    +      <version>19.0</version>
    +    </dependency>
    --- End diff --
   
    fixed


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2148
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5143/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2148
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3921/



---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/2148
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2148
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3924/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2148#discussion_r182636062
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java ---
    @@ -354,7 +365,9 @@ protected Expression getFilterPredicates(Configuration configuration) {
         DataMapJob dataMapJob = getDataMapJob(job.getConfiguration());
         List<PartitionSpec> partitionsToPrune = getPartitionsToPrune(job.getConfiguration());
         List<ExtendedBlocklet> prunedBlocklets;
    -    if (distributedCG || dataMapExprWrapper.getDataMapType() == DataMapLevel.FG) {
    +    if (isFgDataMapPruningEnable(job.getConfiguration()) &&
    +        (distributedCG || dataMapExprWrapper.getDataMapType() == DataMapLevel.FG) &&
    +        dataMapJob != null) {
    --- End diff --
   
    But I guess we should prune with blockletdatamap once if fg is disabled.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2148
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5146/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2148#discussion_r182637982
 
    --- Diff: store/search/src/main/scala/org/apache/spark/rpc/Master.scala ---
    @@ -0,0 +1,215 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.rpc
    +
    +import java.io.IOException
    +import java.net.InetAddress
    +import java.util.{List => JList, Map => JMap, Objects, Random, Set => JSet, UUID}
    +
    +import scala.collection.JavaConverters._
    +import scala.collection.mutable
    +import scala.collection.mutable.ArrayBuffer
    +import scala.concurrent.ExecutionContext.Implicits.global
    --- End diff --
   
    Unused import


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2148
 
    @jackylk But with sparks RPC endpoint framework it is dependent on spark, so is it possible for carbon to have separate cluster without running sparks executors?


---