[GitHub] incubator-carbondata pull request #457: [WIP] Keep only one location when ge...

[qiuchenjian-2]

GitHub user Jay357089 opened a pull request:

    https://github.com/apache/incubator-carbondata/pull/457

    [WIP] Keep only one location when getPreferedLocation in Loading to avoid multiple …

   

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Jay357089/incubator-carbondata keepOneLocation

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/457.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #457
   
----
commit f77d9b9d80dd1d31c87b2cbc5dad1fbaab8d37b2
Author: Jay357089 <[hidden email]>
Date:   2016-12-23T08:01:14Z

    keep only one location when getPrefered in Loading to avoid multiple tasks in one node

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/457
 
    Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/298/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user eason-lyx commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/457
 
    @Jay357089 previous implement multiple preferlocaton in case of one node crush then will retry the task on the same node,can u verify if we just give only one preferlocation, and hack the task failure,check whether the task will always relaunch on the same node.
   
    if spark can choose other node ,then it will be better keep only one preferlocation owing to the data locality.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user Jay357089 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/457
 
     @eason-lyx
    ###  i verifeid in 6 nodes yarn cluster. after i killed coarseGrainExecutorbacked in one node, then the task can be restarted in another node. Process infomation is like below:
    INFO  23-12 07:36:19,212 - Starting task 2.0 in stage 7.0 (TID 316, linux-136, partition 2,NODE_LOCAL, 4240 bytes)
    INFO  23-12 07:36:19,212 - Starting task 3.0 in stage 7.0 (TID 317, linux-134, partition 3,NODE_LOCAL, 4240 bytes)
    INFO  23-12 07:36:19,212 - Starting task 0.0 in stage 7.0 (TID 318, linux-139, partition 0,NODE_LOCAL, 4240 bytes)
    INFO  23-12 07:36:19,212 - Starting task 1.0 in stage 7.0 (TID 319, linux-135, partition 1,NODE_LOCAL, 4274 bytes)
    INFO  23-12 07:36:19,212 - Starting task 5.0 in stage 7.0 (TID 320, linux-141, partition 5,NODE_LOCAL, 4233 bytes)
    INFO  23-12 07:36:19,212 - Starting task 4.0 in stage 7.0 (TID 321, linux-140, partition 4,NODE_LOCAL, 4233 bytes)
   
    ERROR 23-12 07:37:48,021 - Lost executor 1 on linux-139: Container marked as failed: container_1482169114977_0026_01_000002 on host: linux-139. Exit status: 137. Diagnostics: Container killed on request. Exit code is 137
    Container exited with a non-zero exit code 137
    Killed by external signal
   
    INFO  23-12 07:37:48,758 - Starting task 0.1 in stage 7.0 (TID 322, linux-141, partition 0,RACK_LOCAL, 4240 bytes)
   
    ### and the amount of  finally generated carbon data file is same with that generated in normal process.
    ### so, i think it's okay to keep only one preferred location.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user manishgupta88 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/457
 
    @Jay357089 ..I agree with @eason-lyx ...If data load fails on one node then the job will be resubmitted to the same node again. If you kill the executor then spark will submit the job to other executors but consider the case when the data load has failed due to any other reason like **disk space is full**, etc. then in those cases job need to be submitted on the other executors. But according to these changes it wont be submitted.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/457
 
    Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/365/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user Jay357089 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/457
 
    @manishgupta88  yes, i checked in cluster, you are right. but see,  if one node has problem that can not be recovered, then user should resolve  the problem on this node first, otherwise,  he has to tolerate the low load speed.  or, do u have any idea to solve load balance as well as task failure?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user Jay357089 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/457
 
    @jackylk hi, Jacky, what's your opinion?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user manishgupta88 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/457
 
    @Jay357089 ....this is how spark behaves. If a task fails on one node it will submit the same task on the other node considering that there could be some problem with the node. Also this will not be an usual behavior wherein one node will fail always, this is a rare case that will happen. So in my opinion giving multiple options to spark for data load is a good option.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user Jay357089 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/457
 
    @manishgupta88 but the problem is, if we give multiple location, then spark may run multiple tasks in one node, which is bad for load performance. i checked with spark guy, he said that because partitions are few, while cores are many, so the case that multiple tasks in one node  is easy to happen.  Furthermore,  considering that the case you mentioned is rare, why not give only one location ?  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user Jay357089 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/457
 
    close it now, need to find better way.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user Jay357089 closed the pull request at:

    https://github.com/apache/incubator-carbondata/pull/457


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---