[GitHub] carbondata pull request #1038: [WIP][CARBONDATA-1179] Improve the Size calcu...

[qiuchenjian-2]

GitHub user sraghunandan opened a pull request:

    https://github.com/apache/carbondata/pull/1038

    [WIP][CARBONDATA-1179] Improve the Size calculation of Objects being added and managed in LRU cache

   

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sraghunandan/carbondata-1 lru_object_size_calculation

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1038.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1038
   
----
commit db6c074e842d281f22d1c165b7f542835545a4f8
Author: Raghunandan S <[hidden email]>
Date:   2017-06-14T15:31:07Z

    lru object size calculation

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2499/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
   
    Refer to this link for build results (access rights to CI server needed):
    https://builds.apache.org/job/carbondata-pr-spark-1.6/380/<h2>Failed Tests: <span class='status-failure'>3</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/380/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>3</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/380/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.partition/TestDataLoadingForPartitionTable/data_loading_for_partition_table__range_partition/'><strong>org.apache.carbondata.spark.testsuite.partition.TestDataLoadingForPartitionTable.data loading for partition table: range partition</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/380/org.apache.carbondata$carbondata-spa
 rk-common-test/testReport/org.apache.carbondata.spark.testsuite.partition/TestDataLoadingForPartitionTable/Insert_into_for_partition_table__range_partition/'><strong>org.apache.carbondata.spark.testsuite.partition.TestDataLoadingForPartitionTable.Insert into for partition table: range partition</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/380/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.partition/TestQueryForPartitionTable/detail_query_on_partition_table__range_partition/'><strong>org.apache.carbondata.spark.testsuite.partition.TestQueryForPartitionTable.detail query on partition table: range partition</strong></a></li></ul>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user mohammadshahidkhan commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1038#discussion_r122628730
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/AbstractBlockIndexStoreCache.java ---
    @@ -92,13 +94,14 @@ protected void checkAndLoadTableBlocks(AbstractIndex tableBlock,
         TableBlockInfo blockInfo = tableBlockUniqueIdentifier.getTableBlockInfo();
         long requiredMetaSize = CarbonUtil.calculateMetaSize(blockInfo);
    --- End diff --
   
    Better we can remove CarbonUtil.calculateMetaSize(blockInfo)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user mohammadshahidkhan commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1038#discussion_r122636342
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/SegmentTaskIndexStore.java ---
    @@ -233,8 +235,7 @@ private SegmentTaskIndexWrapper loadAndGetTaskIdToSegmentsMap(
                       taskIdToTableBlockInfoMap.entrySet().iterator();
                   long requiredSize =
                       calculateRequiredSize(taskIdToTableBlockInfoMap, absoluteTableIdentifier);
    -              segmentTaskIndexWrapper
    -                  .setMemorySize(requiredSize + segmentTaskIndexWrapper.getMemorySize());
    +
                   boolean isAddedToLruCache =
    --- End diff --
   
    Putting the segment cache object prior to loading the segment b-tree could
    allow dirty read in case of concurrent query.
    Better to add the cache object after finishing the segment load.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user sraghunandan commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1038#discussion_r123767855
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/AbstractBlockIndexStoreCache.java ---
    @@ -92,13 +94,14 @@ protected void checkAndLoadTableBlocks(AbstractIndex tableBlock,
         TableBlockInfo blockInfo = tableBlockUniqueIdentifier.getTableBlockInfo();
         long requiredMetaSize = CarbonUtil.calculateMetaSize(blockInfo);
    --- End diff --
   
    any particular reason? we are using the value to determine whether we need to load the block into memory.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
    Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/128/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2702/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
   
    Refer to this link for build results (access rights to CI server needed):
    https://builds.apache.org/job/carbondata-pr-spark-1.6/627/<h2>Failed Tests: <span class='status-failure'>1</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-core' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/627/org.apache.carbondata$carbondata-core/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-core</a>: <span class='status-failure'>1</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/627/org.apache.carbondata$carbondata-core/testReport/org.apache.carbondata.core.cache.dictionary/ReverseDictionaryCacheTest/testLRUCacheForKeyDeletionAfterMaxSizeIsReached/'><strong>org.apache.carbondata.core.cache.dictionary.ReverseDictionaryCacheTest.testLRUCacheForKeyDeletionAfterMaxSizeIsReached</strong></a></li></ul>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
    Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/160/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2736/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
    Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/161/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2737/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
   
    Refer to this link for build results (access rights to CI server needed):
    https://builds.apache.org/job/carbondata-pr-spark-1.6/666/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
   
    Refer to this link for build results (access rights to CI server needed):
    https://builds.apache.org/job/carbondata-pr-spark-1.6/665/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
    Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/198/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2777/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/1038
 
   
    Refer to this link for build results (access rights to CI server needed):
    https://builds.apache.org/job/carbondata-pr-spark-1.6/707/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1038#discussion_r124599881
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java ---
    @@ -199,6 +199,36 @@ public boolean put(String columnIdentifier, Cacheable cacheInfo, long requiredSi
       }
     
       /**
    +   * This method will check if required size is available in the memory
    +   * @param columnIdentifier
    +   * @param cacheInfo
    +   * @param requiredSize
    +   * @return
    +   */
    +  public boolean tryPut(String columnIdentifier, long requiredSize) {
    +    if (LOGGER.isDebugEnabled()) {
    --- End diff --
   
    It can better remove entry so that temp block memory also with in LRU limit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---