[GitHub] carbondata pull request #1935: [CARBONDATA-2134] Prevent implicit column fil...

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1935: [CARBONDATA-2134] Prevent implicit column fil...

qiuchenjian-2
GitHub user manishgupta88 opened a pull request:

    https://github.com/apache/carbondata/pull/1935

    [CARBONDATA-2134] Prevent implicit column filter list from getting serialized while submitting task to executor

    **Problem**
    In the current store blocklet pruning in driver and no further pruning takes place in the executor side. But still the implicit column filter list being sent to executor. As the size of list grows the cost of serializing and deserializing the list is increasing which can impact the query performance.
   
    **Solution**
    Remove the list from the filter expression before submitting the task to executor.
   
    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed?
     No
     - [ ] Any backward compatibility impacted?
     No
     - [ ] Document update required?
    No
     - [ ] Testing done
    UT added
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
    NA


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/manishgupta88/carbondata executor_filter_list_serialization

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1935.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1935
   
----
commit 4630dbf30f26adff8b59ae57af12a2b183e8195f
Author: m00258959 <manish.gupta@...>
Date:   2018-02-05T11:40:18Z

    Modified code to prevent implicit column array list from serializing and deserializing to executor to improve query performance

----


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1935: [CARBONDATA-2134] Prevent implicit column filter lis...

qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1935
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3535/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1935: [CARBONDATA-2134] Prevent implicit column filter lis...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1935
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2299/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1935: [CARBONDATA-2134] Prevent implicit column filter lis...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1935
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3377/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1935: [CARBONDATA-2134] Prevent implicit column fil...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1935#discussion_r166845170
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java ---
    @@ -1003,4 +1004,13 @@ public static String getTableName(Configuration configuration)
         }
         return tableName;
       }
    +
    +  /**
    +   * Method to remove InExpression node from filter expression
    +   *
    +   * @param expression
    +   */
    +  public void removeInExpressionFromFilterExpression(Expression expression) {
    --- End diff --
   
    This method should not belong here.  Better do in scanrdd only


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1935: [CARBONDATA-2134] Prevent implicit column filter lis...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1935
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3585/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1935: [CARBONDATA-2134] Prevent implicit column filter lis...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1935
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2348/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1935: [CARBONDATA-2134] Prevent implicit column filter lis...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1935
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3434/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1935: [CARBONDATA-2134] Prevent implicit column filter lis...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1935
 
    LGTM


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1935: [CARBONDATA-2134] Prevent implicit column fil...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/1935


---