[GitHub] carbondata pull request #2290: [CARBONDATA-2389] Search mode support lucene ...

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5008/



---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r189522994
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java ---
    @@ -90,13 +90,18 @@ private DataMapStoreManager() {
           String dbName = carbonTable.getDatabaseName();
           String tableName = carbonTable.getTableName();
           String dmName = dataMap.getDataMapSchema().getDataMapName();
    -      boolean isDmVisible = sessionInfo.getSessionParams().getProperty(
    -          String.format("%s%s.%s.%s", CarbonCommonConstants.CARBON_DATAMAP_VISIBLE,
    -              dbName, tableName, dmName), "true").trim().equalsIgnoreCase("true");
    -      if (!isDmVisible) {
    -        LOGGER.warn(String.format("Ignore invisible datamap %s on table %s.%s",
    -            dmName, dbName, tableName));
    -        dataMapIterator.remove();
    +      if (sessionInfo != null) {
    --- End diff --
   
    Yes, there are not carbon session info in search mode. Can we add datamap visible status  into CabonProperties?


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r189598051
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/AndDataMapExprWrapper.java ---
    @@ -59,6 +60,21 @@ public AndDataMapExprWrapper(DataMapExprWrapper left, DataMapExprWrapper right,
         return andBlocklets;
       }
     
    +  @Override
    +  public List<ExtendedBlocklet> prune(DataMapDistributable distributable,
    +                                      List<PartitionSpec> partitionsToPrune)
    +          throws IOException {
    +    List<ExtendedBlocklet> leftPrune = left.prune(distributable, partitionsToPrune);
    +    List<ExtendedBlocklet> rightPrune = right.prune(distributable, partitionsToPrune);
    +    List<ExtendedBlocklet> andBlocklets = new ArrayList<>();
    +    for (ExtendedBlocklet blocklet : leftPrune) {
    +      if (rightPrune.contains(blocklet)) {
    --- End diff --
   
    When will you use org.apache.carbondata.core.datamap.dev.expr.AndDataMapExprWrapper#prune(java.util.List<org.apache.carbondata.core.datamap.Segment>, java.util.List<org.apache.carbondata.core.indexstore.PartitionSpec>) or org.apache.carbondata.core.datamap.dev.expr.AndDataMapExprWrapper#prune(org.apache.carbondata.core.datamap.DataMapDistributable, java.util.List<org.apache.carbondata.core.indexstore.PartitionSpec>)?


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r189598687
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/LatestFilesReadCommittedScope.java ---
    @@ -40,9 +40,26 @@
     public class LatestFilesReadCommittedScope implements ReadCommittedScope {
     
       private String carbonFilePath;
    +  private String segmentId;
       private ReadCommittedIndexFileSnapShot readCommittedIndexFileSnapShot;
       private LoadMetadataDetails[] loadMetadataDetails;
     
    +  /**
    +   * a new constructor of this class, which supports obtain lucene index in search mode
    +   *
    +   * @param path      carbon file path
    +   * @param segmentId segment id
    +   */
    +  public LatestFilesReadCommittedScope(String path, String segmentId) {
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    please remove the CSV in this PR


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6021/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4862/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5034/



---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    @xuchuanyin I am not sure, no plan yet


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r189758718
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/LatestFilesReadCommittedScope.java ---
    @@ -40,9 +40,26 @@
     public class LatestFilesReadCommittedScope implements ReadCommittedScope {
     
       private String carbonFilePath;
    +  private String segmentId;
       private ReadCommittedIndexFileSnapShot readCommittedIndexFileSnapShot;
       private LoadMetadataDetails[] loadMetadataDetails;
     
    +  /**
    +   * a new constructor of this class, which supports obtain lucene index in search mode
    +   *
    +   * @param path      carbon file path
    +   * @param segmentId segment id
    +   */
    +  public LatestFilesReadCommittedScope(String path, String segmentId) {
    --- End diff --
   
    Can not operate like this, because this(path) has takeCarbonIndexFileSnapShot method, which dependency on segmentId.


---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    @jackylk Ok, removed.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6025/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4866/



---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5040/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5041/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6029/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4870/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5043/



---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Please review it. @jackylk @ravipesala @QiangCai  


---