[GitHub] carbondata pull request #2783: [HotFix] use unsafe for query model based on ...

[qiuchenjian-2]

GitHub user xuchuanyin opened a pull request:

    https://github.com/apache/carbondata/pull/2783

    [HotFix] use unsafe for query model based on sytem property

    use unsafe for query model based on sytem property
   
    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
   
     - [ ] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xuchuanyin/carbondata 0928_fix_bloom_jvm_crash

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2783.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2783
   
----
commit 5f3c1ebf6c7c73bf264b312a8de89d7d95665a51
Author: xuchuanyin <xuchuanyin@...>
Date:   2018-09-28T10:37:07Z

    use unsafe for query model based on sytem property
   
    use unsafe for query model based on sytem property

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2783
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/629/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2783#discussion_r221225548
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/model/QueryModel.java ---
    @@ -120,7 +121,7 @@
       private boolean isFG;
     
       // whether to clear/free unsafe memory or not
    -  private boolean freeUnsafeMemory = true;
    +  private boolean freeUnsafeMemory;
    --- End diff --
   
    suggest to remove this variable, use system property whenever it is needed


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2783
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8890/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2783
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/821/



---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2783#discussion_r221237767
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/model/QueryModel.java ---
    @@ -120,7 +121,7 @@
       private boolean isFG;
     
       // whether to clear/free unsafe memory or not
    -  private boolean freeUnsafeMemory = true;
    +  private boolean freeUnsafeMemory;
    --- End diff --
   
    oh, that will need many changes.
    I think it's OK to get this value from system property at the beginning and pass the value to the place where it is needed, cause the system property can be modified during processing. And in this scenario, directly using system property will cause problem.


---

[qiuchenjian-2]

Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/2783
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2783
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/635/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2783
 
    @xuchuanyin I don't what is the purpose of doing this? Please elaborate problem in PR description.
    This variable is added to avoid JVM crash when freed multiple times. I am not getting why this should be configurable.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2783
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/830/



---

[qiuchenjian-2]

Github user kumarvishal09 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2783#discussion_r221266728
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/model/QueryModel.java ---
    @@ -120,7 +121,7 @@
       private boolean isFG;
     
       // whether to clear/free unsafe memory or not
    -  private boolean freeUnsafeMemory = true;
    +  private boolean freeUnsafeMemory;
    --- End diff --
   
    @xuchuanyin I think you are not clear on the purpose of this variable.
   
    After each task completion we are freeing all memory allocated for task to avoid memory leak based on the task id.
    In case of 'insert into' scenario, data load and query both will have same task id as they run on same thread.
    so as query will finish first, it was clearing all memory (load memory also). so as data loading was is in progress it was causing jvm crash. For this issue we added this variable to avoid freeing memory from query flow in case of insert into.
   
    Also end-user should not have this level control. What was the problem faced ? Why decided to make it a system property ?


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2783
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8900/



---

[qiuchenjian-2]

Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/2783
 
    @ravipesala @kumarvishal09 We have encountered JVM crash problem.
    The dump file complains
    
   
    Clearly it is unsafe memory related problem. Since in the previous PR, we have disabled the unsafe for query, I think in this place we should disable the unsafe accordingly.
   



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2783
 
    @xuchuanyin This problem is fixed in PR https://github.com/apache/carbondata/pull/2793


---

[qiuchenjian-2]

Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/2783
 
    OK, I'll close this PR instead


---

[qiuchenjian-2]

Github user xuchuanyin closed the pull request at:

    https://github.com/apache/carbondata/pull/2783


---