[GitHub] carbondata pull request #2883: [CARBONDATA-3062] Fix Compatibility issue wit...

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2883: [CARBONDATA-3062] Fix Compatibility issue wit...

qiuchenjian-2
GitHub user Indhumathi27 opened a pull request:

    https://github.com/apache/carbondata/pull/2883

    [CARBONDATA-3062] Fix Compatibility issue with cache_level as blocklet

       
    **Why this PR for?**
    In case of hybrid store we can have block as well as blocklet schema.
    Scenario:
    When there is a hybrid store in which few loads are from legacy store which do not contain the blocklet information and hence they will be, by default have cache_level as BLOCK and few loads with latest store which contain the BLOCKLET information and have cache_level BLOCKLET. For these type of scenarios we need to have separate task and footer schemas. For all loads with/without blocklet info there will not be any additional cost of maintaining 2 variables
   
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
   
     - [ ] Testing done
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Indhumathi27/carbondata column_comp

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2883.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2883
   
----

----


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2883: [CARBONDATA-3062] Fix Compatibility issue with cache...

qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2883
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1168/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2883: [CARBONDATA-3062] Fix Compatibility issue with cache...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2883
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1381/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2883: [CARBONDATA-3062] Fix Compatibility issue with cache...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2883
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9432/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2883: [CARBONDATA-3062] Fix Compatibility issue with cache...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2883
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1169/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2883: [CARBONDATA-3062] Fix Compatibility issue with cache...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2883
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1382/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2883: [CARBONDATA-3062] Fix Compatibility issue with cache...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2883
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9433/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2883: [CARBONDATA-3062] Fix Compatibility issue wit...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user manishgupta88 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2883#discussion_r229628306
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/block/SegmentPropertiesAndSchemaHolder.java ---
    @@ -350,47 +358,77 @@ public SegmentProperties getSegmentProperties() {
           return columnCardinality;
         }
     
    -    public CarbonRowSchema[] getTaskSummarySchema(boolean storeBlockletCount,
    +    public CarbonRowSchema[] getTaskSummarySchemaForBlock(boolean storeBlockletCount,
             boolean filePathToBeStored) throws MemoryException {
    -      if (null == taskSummarySchema) {
    +      if (null == taskSummarySchemaForBlock) {
             synchronized (taskSchemaLock) {
    -          if (null == taskSummarySchema) {
    -            taskSummarySchema = SchemaGenerator
    +          if (null == taskSummarySchemaForBlock) {
    +            taskSummarySchemaForBlock = SchemaGenerator
                     .createTaskSummarySchema(segmentProperties, minMaxCacheColumns, storeBlockletCount,
                         filePathToBeStored);
               }
             }
           }
    -      return taskSummarySchema;
    +      return taskSummarySchemaForBlock;
    +    }
    +
    +    public CarbonRowSchema[] getTaskSummarySchemaForBlocklet(boolean storeBlockletCount,
    +        boolean filePathToBeStored) throws MemoryException {
    +      if (null == taskSummarySchemaForBlocklet) {
    +        synchronized (taskSchemaLock) {
    +          if (null == taskSummarySchemaForBlocklet) {
    +            taskSummarySchemaForBlocklet = SchemaGenerator
    +                .createTaskSummarySchema(segmentProperties, minMaxCacheColumns, storeBlockletCount,
    +                    filePathToBeStored);
    +          }
    +        }
    +      }
    +      return taskSummarySchemaForBlocklet;
         }
     
         public CarbonRowSchema[] getBlockFileFooterEntrySchema() {
    -      return getOrCreateFileFooterEntrySchema(true);
    +      if (null == fileFooterEntrySchemaForBlock) {
    +        synchronized (fileFooterSchemaLock) {
    +          if (null == fileFooterEntrySchemaForBlock) {
    +            fileFooterEntrySchemaForBlock =
    +                SchemaGenerator.createBlockSchema(segmentProperties, minMaxCacheColumns);
    +          }
    +        }
    +      }
    +      return fileFooterEntrySchemaForBlock;
         }
     
         public CarbonRowSchema[] getBlockletFileFooterEntrySchema() {
    -      return getOrCreateFileFooterEntrySchema(false);
    +      if (null == fileFooterEntrySchemaForBlocklet) {
    +        synchronized (fileFooterSchemaLock) {
    +          if (null == fileFooterEntrySchemaForBlocklet) {
    +            fileFooterEntrySchemaForBlocklet =
    +                SchemaGenerator.createBlockletSchema(segmentProperties, minMaxCacheColumns);
    +          }
    +        }
    +      }
    +      return fileFooterEntrySchemaForBlocklet;
         }
     
         public List<CarbonColumn> getMinMaxCacheColumns() {
           return minMaxCacheColumns;
         }
     
         private CarbonRowSchema[] getOrCreateFileFooterEntrySchema(boolean isCacheLevelBlock) {
    --- End diff --
   
    Remove this method it is not required


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2883: [CARBONDATA-3062] Fix Compatibility issue with cache...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2883
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1397/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2883: [CARBONDATA-3062] Fix Compatibility issue with cache...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2883
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1184/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2883: [CARBONDATA-3062] Fix Compatibility issue with cache...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2883
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1404/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2883: [CARBONDATA-3062] Fix Compatibility issue with cache...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2883
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9450/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2883: [CARBONDATA-3062] Fix Compatibility issue with cache...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2883
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1190/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2883: [CARBONDATA-3062] Fix Compatibility issue with cache...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user manishgupta88 commented on the issue:

    https://github.com/apache/carbondata/pull/2883
 
    LGTM


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2883: [CARBONDATA-3062] Fix Compatibility issue wit...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/2883


---