[GitHub] carbondata pull request #1242: [WIP]Offheap sort size validation

[qiuchenjian-2]

GitHub user sraghunandan opened a pull request:

    https://github.com/apache/carbondata/pull/1242

    [WIP]Offheap sort size validation

   

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sraghunandan/carbondata-1 offheap_sort_size_validation

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1242.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1242
   
----
commit 69220fce0a4e2196e0a094ac3a0f5c5fb4013bd5
Author: sraghunandan <[hidden email]>
Date:   2017-08-07T01:38:45Z

    Request 1 executor when preferred locations are empty

commit 8b7027ac98090dcd799f2e91feb106ef3a02915a
Author: sraghunandan <[hidden email]>
Date:   2017-08-07T02:03:29Z

    Validate off heap sort size value configured is in valid range

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/1242
 
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/1242
 
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user zzcclp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1242#discussion_r131576451
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/hive/DistributionUtil.scala ---
    @@ -115,7 +115,9 @@ object DistributionUtil {
           sparkContext: SparkContext): Seq[String] = {
         val confExecutors: Int = getConfiguredExecutors(sparkContext)
         LOGGER.info(s"Executors configured : $confExecutors")
    -    val requiredExecutors = if (nodesOfData < 1 || nodesOfData > confExecutors) {
    +    val requiredExecutors = if (nodesOfData < 1) {
    +      1
    +    } else if (nodesOfData > confExecutors) {
    --- End diff --
   
    this change duplicates [PR-1241](https://github.com/apache/carbondata/pull/1241)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1242
 
    I think validation already present. `CarbonProperties.getSortMemoryChunkSizeInMB`. It is duplicated


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1242
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3404/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1242
 
    SDV Build Success with Spark 2.1, Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/132/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1242
 
    Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/806/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user sraghunandan commented on the issue:

    https://github.com/apache/carbondata/pull/1242
 
    retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1242
 
    SDV Build Success with Spark 2.1, Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/230/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user sraghunandan commented on the issue:

    https://github.com/apache/carbondata/pull/1242
 
    retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user sraghunandan commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1242#discussion_r135423319
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/hive/DistributionUtil.scala ---
    @@ -115,7 +115,9 @@ object DistributionUtil {
           sparkContext: SparkContext): Seq[String] = {
         val confExecutors: Int = getConfiguredExecutors(sparkContext)
         LOGGER.info(s"Executors configured : $confExecutors")
    -    val requiredExecutors = if (nodesOfData < 1 || nodesOfData > confExecutors) {
    +    val requiredExecutors = if (nodesOfData < 1) {
    +      1
    +    } else if (nodesOfData > confExecutors) {
    --- End diff --
   
    This is worked over 1241.When 1241 is merged, this change will not reflect


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user sraghunandan closed the pull request at:

    https://github.com/apache/carbondata/pull/1242


---