[GitHub] carbondata pull request #1692: [CARBONDATA-1777] Added check to update relat...

[qiuchenjian-2]

GitHub user kunal642 opened a pull request:

    https://github.com/apache/carbondata/pull/1692

    [CARBONDATA-1777] Added check to update relation if catalog relation is present in plan

    **Analysis:** In spark 2.2 while doing lookup relation there was no case to handle CatalogRelation due to which the tables were not getting refreshed in different sessions.
   
    **Solution:** Add case for CatalogRelation so that the table being referred is refreshed.
   
    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed?
     No
     - [ ] Any backward compatibility impacted?
     No
     - [ ] Document update required?
    No
     - [ ] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
    No


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kunal642/carbondata table_refresh_fix

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1692.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1692
   
----
commit 099f9806bb16c6dbaa747316adca02028b323686
Author: kunal642 <kunalkapoor642@...>
Date:   2017-12-20T08:35:06Z

    added check to update relation if catalog relation is present in plan

----


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2456/



---

[qiuchenjian-2]

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    retest this please


---

[qiuchenjian-2]

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    retest this please


---

[qiuchenjian-2]

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    retest sdv please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2234/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1010/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2493/



---

[qiuchenjian-2]

Github user kunal642 commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2250/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1027/



---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1692#discussion_r158455641
 
    --- Diff: integration/spark2/src/main/spark2.2/CarbonSessionState.scala ---
    @@ -20,6 +20,7 @@ package org.apache.spark.sql.hive
     import scala.collection.generic.SeqFactory
     
     import org.apache.hadoop.conf.Configuration
    +import org.apache.spark.SPARK_VERSION
    --- End diff --
   
    Remove Spark Version


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1692#discussion_r158457326
 
    --- Diff: integration/spark2/src/main/spark2.2/CarbonSessionState.scala ---
    @@ -121,6 +123,19 @@ class CarbonSessionCatalog(
             toRefreshRelation = refreshRelationFromCache(name, carbonDatasourceHadoopRelation)
           case LogicalRelation(carbonDatasourceHadoopRelation: CarbonDatasourceHadoopRelation, _, _) =>
             toRefreshRelation = refreshRelationFromCache(name, carbonDatasourceHadoopRelation)
    +      case SubqueryAlias(_, relation) if
    +      relation.getClass.getName.equals("org.apache.spark.sql.catalyst.catalog.CatalogRelation") ||
    +      relation.getClass.getName.equals("org.apache.spark.sql.catalyst.catalog.HiveTableRelation") ||
    +      relation.getClass.getName.equals(
    +        "org.apache.spark.sql.catalyst.catalog.UnresolvedCatalogRelation") =>
    +        val catalogTable = CarbonReflectionUtils.getFieldOfCatalogTable("tableMeta",
    +          relation).asInstanceOf[CatalogTable]
    +        carbonEnv.carbonMetastore.checkSchemasModifiedTimeAndReloadTable(catalogTable.identifier)
    --- End diff --
   
    In case the table has different timestamp and removed from metadata through checkSchemasModifiedTimeAndReloadTable, then only call refresh table.


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1692#discussion_r158457994
 
    --- Diff: integration/spark2/src/main/spark2.2/CarbonSessionState.scala ---
    @@ -121,6 +123,19 @@ class CarbonSessionCatalog(
             toRefreshRelation = refreshRelationFromCache(name, carbonDatasourceHadoopRelation)
           case LogicalRelation(carbonDatasourceHadoopRelation: CarbonDatasourceHadoopRelation, _, _) =>
             toRefreshRelation = refreshRelationFromCache(name, carbonDatasourceHadoopRelation)
    +      case SubqueryAlias(_, relation) if
    +      relation.getClass.getName.equals("org.apache.spark.sql.catalyst.catalog.CatalogRelation") ||
    +      relation.getClass.getName.equals("org.apache.spark.sql.catalyst.catalog.HiveTableRelation") ||
    +      relation.getClass.getName.equals(
    +        "org.apache.spark.sql.catalyst.catalog.UnresolvedCatalogRelation") =>
    +        val catalogTable = CarbonReflectionUtils.getFieldOfCatalogTable("tableMeta",
    +          relation).asInstanceOf[CatalogTable]
    +        carbonEnv.carbonMetastore.checkSchemasModifiedTimeAndReloadTable(catalogTable.identifier)
    +        refreshTable(catalogTable.identifier)
    +        DataMapStoreManager.getInstance().
    --- End diff --
   
    Please add a log to inform if the table got refreshed.


---

[qiuchenjian-2]

Github user gvramana commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    LGTM


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2520/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2273/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2523/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1052/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1692
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1062/



---