[GitHub] carbondata pull request #2255: [CARBONDATA-2416] Support DEFERRED REBUILD wh...

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2255#discussion_r186350293
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala ---
    @@ -110,6 +110,14 @@ class CarbonSession(@transient val sc: SparkContext,
         )
       }
     
    +  /**
    +   * Return true if the specified sql statement will hit the datamap
    +   */
    +  def isDataMapHit(sqlStatement: String, dataMapName: String): Boolean = {
    --- End diff --
   
    fixed


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2255#discussion_r186351682
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java ---
    @@ -431,7 +432,9 @@ protected Expression getFilterPredicates(Configuration configuration) {
         // Get the available CG datamaps and prune further.
         DataMapExprWrapper cgDataMapExprWrapper = DataMapChooser.get()
             .chooseCGDataMap(getOrCreateCarbonTable(job.getConfiguration()), resolver);
    -    if (cgDataMapExprWrapper != null) {
    +    if (cgDataMapExprWrapper != null &&
    +        DataMapStatusManager.isDataMapEnabled(
    +            cgDataMapExprWrapper.getDataMapSchema().getDataMapName())) {
    --- End diff --
   
    logic modified, please check again


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2255#discussion_r186351820
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala ---
    @@ -110,6 +110,14 @@ class CarbonSession(@transient val sc: SparkContext,
         )
       }
     
    +  /**
    +   * Return true if the specified sql statement will hit the datamap
    +   */
    +  def isDataMapHit(sqlStatement: String, dataMapName: String): Boolean = {
    --- End diff --
   
    I have added an annotation for it, I think it is useful in multiple test suite, so add it here


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2255#discussion_r186355275
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/status/DataMapStatusManager.java ---
    @@ -51,6 +51,16 @@ private DataMapStatusManager() {
         return storageProvider.getDataMapStatusDetails();
       }
     
    +  public static boolean isDataMapEnabled(String dataMapName) throws IOException {
    --- End diff --
   
    ok. will be fixed


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2255#discussion_r186355295
 
    --- Diff: integration/spark2/src/test/scala/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMapSuite.scala ---
    @@ -41,25 +42,34 @@ class BloomCoarseGrainDataMapSuite extends QueryTest with BeforeAndAfterAll {
         sql(s"DROP TABLE IF EXISTS $bloomDMSampleTable")
       }
     
    -  private def checkQuery = {
    +  override def afterEach(): Unit = {
    +    sql(s"DROP TABLE IF EXISTS $normalTable")
    +    sql(s"DROP TABLE IF EXISTS $bloomDMSampleTable")
    +  }
    +
    +  private def dmSql(sqlText: String, dataMapName: String, shouldHit: Boolean): DataFrame = {
    --- End diff --
   
    fixed


---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    @xuchuanyin With `DEFERED REBUILD` syntax, it means the datamap loading will be lazy. If there are new load to the fact table, this datamap will be disabled until user trigger `REBUILD DATAMAP`.



---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    all comment are handled


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4533/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5693/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4771/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4573/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5742/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4802/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4804/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4587/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4811/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4812/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5750/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4591/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2255
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5761/



---