GitHub user Indhumathi27 opened a pull request:
https://github.com/apache/carbondata/pull/2883 [CARBONDATA-3062] Fix Compatibility issue with cache_level as blocklet **Why this PR for?** In case of hybrid store we can have block as well as blocklet schema. Scenario: When there is a hybrid store in which few loads are from legacy store which do not contain the blocklet information and hence they will be, by default have cache_level as BLOCK and few loads with latest store which contain the BLOCKLET information and have cache_level BLOCKLET. For these type of scenarios we need to have separate task and footer schemas. For all loads with/without blocklet info there will not be any additional cost of maintaining 2 variables - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata column_comp Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2883.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2883 ---- ---- --- |
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2883 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1168/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2883 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1381/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2883 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9432/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2883 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1169/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2883 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1382/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2883 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9433/ --- |
In reply to this post by qiuchenjian-2
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2883#discussion_r229628306 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/block/SegmentPropertiesAndSchemaHolder.java --- @@ -350,47 +358,77 @@ public SegmentProperties getSegmentProperties() { return columnCardinality; } - public CarbonRowSchema[] getTaskSummarySchema(boolean storeBlockletCount, + public CarbonRowSchema[] getTaskSummarySchemaForBlock(boolean storeBlockletCount, boolean filePathToBeStored) throws MemoryException { - if (null == taskSummarySchema) { + if (null == taskSummarySchemaForBlock) { synchronized (taskSchemaLock) { - if (null == taskSummarySchema) { - taskSummarySchema = SchemaGenerator + if (null == taskSummarySchemaForBlock) { + taskSummarySchemaForBlock = SchemaGenerator .createTaskSummarySchema(segmentProperties, minMaxCacheColumns, storeBlockletCount, filePathToBeStored); } } } - return taskSummarySchema; + return taskSummarySchemaForBlock; + } + + public CarbonRowSchema[] getTaskSummarySchemaForBlocklet(boolean storeBlockletCount, + boolean filePathToBeStored) throws MemoryException { + if (null == taskSummarySchemaForBlocklet) { + synchronized (taskSchemaLock) { + if (null == taskSummarySchemaForBlocklet) { + taskSummarySchemaForBlocklet = SchemaGenerator + .createTaskSummarySchema(segmentProperties, minMaxCacheColumns, storeBlockletCount, + filePathToBeStored); + } + } + } + return taskSummarySchemaForBlocklet; } public CarbonRowSchema[] getBlockFileFooterEntrySchema() { - return getOrCreateFileFooterEntrySchema(true); + if (null == fileFooterEntrySchemaForBlock) { + synchronized (fileFooterSchemaLock) { + if (null == fileFooterEntrySchemaForBlock) { + fileFooterEntrySchemaForBlock = + SchemaGenerator.createBlockSchema(segmentProperties, minMaxCacheColumns); + } + } + } + return fileFooterEntrySchemaForBlock; } public CarbonRowSchema[] getBlockletFileFooterEntrySchema() { - return getOrCreateFileFooterEntrySchema(false); + if (null == fileFooterEntrySchemaForBlocklet) { + synchronized (fileFooterSchemaLock) { + if (null == fileFooterEntrySchemaForBlocklet) { + fileFooterEntrySchemaForBlocklet = + SchemaGenerator.createBlockletSchema(segmentProperties, minMaxCacheColumns); + } + } + } + return fileFooterEntrySchemaForBlocklet; } public List<CarbonColumn> getMinMaxCacheColumns() { return minMaxCacheColumns; } private CarbonRowSchema[] getOrCreateFileFooterEntrySchema(boolean isCacheLevelBlock) { --- End diff -- Remove this method it is not required --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2883 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1397/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2883 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1184/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2883 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1404/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2883 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9450/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2883 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1190/ --- |
In reply to this post by qiuchenjian-2
Github user manishgupta88 commented on the issue:
https://github.com/apache/carbondata/pull/2883 LGTM --- |
In reply to this post by qiuchenjian-2
|
Free forum by Nabble | Edit this page |