[GitHub] carbondata pull request #2269: [WIP][LUCENE]close the lucene index reader af...

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4578/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5799/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4645/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4852/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4688/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5844/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4890/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5850/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4694/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4895/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5899/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4933/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5909/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4755/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4942/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2269#discussion_r189220225
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/DistributableDataMapFormat.java ---
    @@ -100,14 +103,18 @@ private static FilterResolverIntf getFilterExp(Configuration configuration) thro
         return new RecordReader<Void, ExtendedBlocklet>() {
           private Iterator<ExtendedBlocklet> blockletIterator;
           private ExtendedBlocklet currBlocklet;
    +      private List<DataMap> dataMaps;
     
           @Override public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptContext)
               throws IOException, InterruptedException {
    -        DataMapDistributableWrapper distributable = (DataMapDistributableWrapper) inputSplit;
    -        TableDataMap dataMap = DataMapStoreManager.getInstance()
    +        distributable = (DataMapDistributableWrapper) inputSplit;
    +        TableDataMap tableDataMap = DataMapStoreManager.getInstance()
                 .getDataMap(table, distributable.getDistributable().getDataMapSchema());
    -        List<ExtendedBlocklet> blocklets = dataMap.prune(distributable.getDistributable(),
    -            dataMapExprWrapper.getFilterResolverIntf(distributable.getUniqueId()), partitions);
    +        dataMaps = tableDataMap.getTableDataMaps(distributable.getDistributable());
    --- End diff --
   
    I don't see the benefit of get the datamaps out and close it, I think you can close inside datamap when prune is done. I think it is not required to change theinterface for it.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4813/



---

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2269#discussion_r189267374
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/DistributableDataMapFormat.java ---
    @@ -100,14 +103,18 @@ private static FilterResolverIntf getFilterExp(Configuration configuration) thro
         return new RecordReader<Void, ExtendedBlocklet>() {
           private Iterator<ExtendedBlocklet> blockletIterator;
           private ExtendedBlocklet currBlocklet;
    +      private List<DataMap> dataMaps;
     
           @Override public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptContext)
               throws IOException, InterruptedException {
    -        DataMapDistributableWrapper distributable = (DataMapDistributableWrapper) inputSplit;
    -        TableDataMap dataMap = DataMapStoreManager.getInstance()
    +        distributable = (DataMapDistributableWrapper) inputSplit;
    +        TableDataMap tableDataMap = DataMapStoreManager.getInstance()
                 .getDataMap(table, distributable.getDistributable().getDataMapSchema());
    -        List<ExtendedBlocklet> blocklets = dataMap.prune(distributable.getDistributable(),
    -            dataMapExprWrapper.getFilterResolverIntf(distributable.getUniqueId()), partitions);
    +        dataMaps = tableDataMap.getTableDataMaps(distributable.getDistributable());
    --- End diff --
   
    on task failure, or if prune fails, then it will not be closed if we try to close after prune, so taskCompletionListener will take care all right?


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5971/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2269
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4995/



---