[GitHub] carbondata pull request #1199: [CARBONDATA-1335] Remove duplicated time-cons...

[qiuchenjian-2]

GitHub user xuchuanyin opened a pull request:

    https://github.com/apache/carbondata/pull/1199

    [CARBONDATA-1335] Remove duplicated time-consuming method call

    # Scenario
   
    Currently we did a concurrent  14 queries on Carbondata. The queries are the same, but on different tables. We have noticed the following scene:
   
    + A single query took about 5s;
    + In concurrent scenario, each query took about 15s;
   
    By adding checkpoint in the log, we found that there was great latency in starting query jobs in spark.
   
    # Analysts
   
    When we fire a query, Carbondata firstly do some job in the client side, including parse/analyze plans and prepare filtered blocks and inputSplits. Then Carbondata start to submit query job to spark.
   
    We found in the first step, Carbondata took about 7s in current scenario, but it only took about <1s in single scenario.
    By studying the related code, we found the most time consuming method call was  `CarbonSessionCatalog.lookupRelation`. In side this method, it called `super.lookupRelation` twice, which consumed about 3s each time.
   
    # Solution
   
    Carbondata only needs to call the `super.lookupRelation` only once, we need to remove the useless duplicated method call.
   
    I've tested in my environment and it works well. In concurrent scenario, each query takes about 12s (3s saved for the improvement).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xuchuanyin/carbondata remove_duplicated_lookup

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1199.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1199
   
----
commit d05d1fadbe0f773a00ff1d0e96ff9fe90b7b7f06
Author: xuchuanyin <[hidden email]>
Date:   2017-07-27T07:07:25Z

    Remove duplicated time-consuming method call

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/1199
 
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/1199
 
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1199#discussion_r129767934
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonSessionState.scala ---
    @@ -106,7 +109,6 @@ class CarbonSessionCatalog(
             }
           case _ =>
         }
    -    super.lookupRelation(name, alias)
    --- End diff --
   
    this PR mainly focus on removing this useless method call


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1199
 
    Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/617/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1199
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3212/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/1199
 
    LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/1199


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---