[GitHub] carbondata pull request #2290: [CARBONDATA-2389] Search mode support lucene ...

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r187768530
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/AndDataMapExprWrapper.java ---
    @@ -59,6 +60,21 @@ public AndDataMapExprWrapper(DataMapExprWrapper left, DataMapExprWrapper right,
         return andBlocklets;
       }
     
    +  @Override
    +  public List<ExtendedBlocklet> prune(DataMapDistributable distributable,
    +                                      List<PartitionSpec> partitionsToPrune)
    +          throws IOException {
    +    List<ExtendedBlocklet> leftPrune = left.prune(distributable, partitionsToPrune);
    +    List<ExtendedBlocklet> rightPrune = right.prune(distributable, partitionsToPrune);
    +    List<ExtendedBlocklet> andBlocklets = new ArrayList<>();
    +    for (ExtendedBlocklet blocklet : leftPrune) {
    +      if (rightPrune.contains(blocklet)) {
    --- End diff --
   
    Have you ever validate the correctness of this? The `equals` and `hashCode` method in `ExtendedBlocklet` only make use of its member `segmentId`. I'm not sure whether it is correct.


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r187768650
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/scanner/impl/BlockletFilterScanner.java ---
    @@ -169,6 +169,10 @@ private BlockletScannedResult executeFilter(RawBlockletColumnChunks rawBlockletC
         // apply filter on actual data, for each page
         BitSetGroup bitSetGroup = this.filterExecuter.applyFilter(rawBlockletColumnChunks,
             useBitSetPipeLine);
    +    // if bitSetGroup is nul, then new BitSetGroup object, which can avoid NPE
    --- End diff --
   
    Is `Lucene` will introduce this problem as you described or is it a bug caused by other scenario?


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r187768616
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/LatestFilesReadCommittedScope.java ---
    @@ -40,9 +40,26 @@
     public class LatestFilesReadCommittedScope implements ReadCommittedScope {
     
       private String carbonFilePath;
    +  private String segmentId;
       private ReadCommittedIndexFileSnapShot readCommittedIndexFileSnapShot;
       private LoadMetadataDetails[] loadMetadataDetails;
     
    +  /**
    +   * a new constructor of this class, which supports obtain lucene index in search mode
    +   *
    +   * @param path      carbon file path
    +   * @param segmentId segment id
    +   */
    +  public LatestFilesReadCommittedScope(String path, String segmentId) {
    --- End diff --
   
    too much duplicate codes there. It can internally call `this(path)` to reduce the code


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r187768472
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/AndDataMapExprWrapper.java ---
    @@ -59,6 +60,21 @@ public AndDataMapExprWrapper(DataMapExprWrapper left, DataMapExprWrapper right,
         return andBlocklets;
       }
     
    +  @Override
    +  public List<ExtendedBlocklet> prune(DataMapDistributable distributable,
    --- End diff --
   
    Is the indent correct?


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r187768465
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java ---
    @@ -90,13 +90,18 @@ private DataMapStoreManager() {
           String dbName = carbonTable.getDatabaseName();
           String tableName = carbonTable.getTableName();
           String dmName = dataMap.getDataMapSchema().getDataMapName();
    -      boolean isDmVisible = sessionInfo.getSessionParams().getProperty(
    -          String.format("%s%s.%s.%s", CarbonCommonConstants.CARBON_DATAMAP_VISIBLE,
    -              dbName, tableName, dmName), "true").trim().equalsIgnoreCase("true");
    -      if (!isDmVisible) {
    -        LOGGER.warn(String.format("Ignore invisible datamap %s on table %s.%s",
    -            dmName, dbName, tableName));
    -        dataMapIterator.remove();
    +      if (sessionInfo != null) {
    --- End diff --
   
    When will sessionInfo be null?


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r187768562
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapExprWrapper.java ---
    @@ -40,6 +41,18 @@
       List<ExtendedBlocklet> prune(List<Segment> segments, List<PartitionSpec> partitionsToPrune)
           throws IOException;
     
    +  /**
    +   * prune blocklet according distributable
    +   *
    +   * @param distributable     distributable
    +   * @param partitionsToPrune partitions to prune
    +   * @return the pruned ExtendedBlocklet list
    +   * @throws IOException
    +   */
    +  List<ExtendedBlocklet> prune(DataMapDistributable distributable,
    --- End diff --
   
    there are other indent problems like this in this PR, better to optimize them all


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r187768741
 
    --- Diff: store/search/src/main/scala/org/apache/spark/rpc/Master.scala ---
    @@ -234,7 +232,7 @@ class Master(sparkConf: SparkConf) {
           // if we have enough data already, we do not need to collect more result
           if (rowCount < globalLimit) {
             // wait for worker for 10s
    --- End diff --
   
    please modify the comments as well


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r187768550
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/expr/DataMapExprWrapper.java ---
    @@ -40,6 +41,18 @@
       List<ExtendedBlocklet> prune(List<Segment> segments, List<PartitionSpec> partitionsToPrune)
           throws IOException;
     
    +  /**
    +   * prune blocklet according distributable
    +   *
    +   * @param distributable     distributable
    +   * @param partitionsToPrune partitions to prune
    +   * @return the pruned ExtendedBlocklet list
    +   * @throws IOException
    +   */
    +  List<ExtendedBlocklet> prune(DataMapDistributable distributable,
    --- End diff --
   
    the indent problem?


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r187768680
 
    --- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneDataMapFactoryBase.java ---
    @@ -168,7 +169,8 @@ public DataMapBuilder createBuilder(Segment segment, String shardName) {
             getAllIndexDirs(tableIdentifier.getTablePath(), segment.getSegmentNo());
         for (CarbonFile indexDir : indexDirs) {
           // Filter out the tasks which are filtered through CG datamap.
    -      if (!segment.getFilteredIndexShardNames().contains(indexDir.getName())) {
    +      if (getDataMapLevel() != DataMapLevel.FG &&
    --- End diff --
   
    What does this for?
    If it is only for CG datamap, then you can judge outside this loop.


---

[qiuchenjian-2]

Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Hi, @xubo245 Will you continue working on make search mode using other types of datamap?


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5855/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4701/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4899/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5857/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4901/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4703/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4902/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4705/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2290
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5859/



---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2290#discussion_r187871019
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java ---
    @@ -90,13 +90,18 @@ private DataMapStoreManager() {
           String dbName = carbonTable.getDatabaseName();
           String tableName = carbonTable.getTableName();
           String dmName = dataMap.getDataMapSchema().getDataMapName();
    -      boolean isDmVisible = sessionInfo.getSessionParams().getProperty(
    -          String.format("%s%s.%s.%s", CarbonCommonConstants.CARBON_DATAMAP_VISIBLE,
    -              dbName, tableName, dmName), "true").trim().equalsIgnoreCase("true");
    -      if (!isDmVisible) {
    -        LOGGER.warn(String.format("Ignore invisible datamap %s on table %s.%s",
    -            dmName, dbName, tableName));
    -        dataMapIterator.remove();
    +      if (sessionInfo != null) {
    --- End diff --
   
    Session info will be null when use the search mode, Searcher=>handleRequest=>DataMapChooser=>getAllVisibleDataMap


---