[GitHub] incubator-carbondata pull request #631: [WIP]Adding Header And Making Footer...

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/631
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1039/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r104914302
 
    --- Diff: format/src/main/thrift/carbondata.thrift ---
    @@ -183,11 +183,23 @@ struct FileFooter{
     }
     
     /**
    +* Footer for indexed carbon file
    +*/
    +struct FileFooterVersion3{
    +    1: required i64 num_rows; // Total number of rows in this file
    +    2: required SegmentInfo segment_info; // Segment info (will be same/repeated for all files in this segment)
    +    3: required list<BlockletIndex> blocklet_index_list; // blocklet index of all blocklets in this file
    +    4: optional list<BlockletInfo3> blocklet_info_list3; // Information about blocklets of all columns in this file
    +    5: optional dictionary.ColumnDictionaryChunk dictionary; // blocklet local dictionary
    +}
    +
    +/**
      * Header for appendable carbon file
      */
     struct FileHeader{
      1: required i32 version; // version used for data compatibility
      2: required list<schema.ColumnSchema> table_columns;  // Description of columns in this file
    + 3: optional bool is_Footer_Present; //  to check whether footer is present or not      
    --- End diff --
   
    please remove `MutationType`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r104914699
 
    --- Diff: format/src/main/thrift/carbondata.thrift ---
    @@ -183,11 +183,23 @@ struct FileFooter{
     }
     
     /**
    +* Footer for indexed carbon file
    +*/
    +struct FileFooterVersion3{
    +    1: required i64 num_rows; // Total number of rows in this file
    +    2: required SegmentInfo segment_info; // Segment info (will be same/repeated for all files in this segment)
    +    3: required list<BlockletIndex> blocklet_index_list; // blocklet index of all blocklets in this file
    +    4: optional list<BlockletInfo3> blocklet_info_list3; // Information about blocklets of all columns in this file
    +    5: optional dictionary.ColumnDictionaryChunk dictionary; // blocklet local dictionary
    +}
    +
    +/**
      * Header for appendable carbon file
      */
     struct FileHeader{
      1: required i32 version; // version used for data compatibility
      2: required list<schema.ColumnSchema> table_columns;  // Description of columns in this file
    --- End diff --
   
    rename to `column_schema`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r104915366
 
    --- Diff: format/src/main/thrift/carbondata.thrift ---
    @@ -183,11 +183,23 @@ struct FileFooter{
     }
     
     /**
    +* Footer for indexed carbon file
    +*/
    +struct FileFooterVersion3{
    --- End diff --
   
    After adding this new struct for V3 format, following is not required in `FileFooter`
    `8: optional list<BlockletInfo3> blocklet_info_list3;`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r104915387
 
    --- Diff: format/src/main/thrift/carbondata.thrift ---
    @@ -183,11 +183,23 @@ struct FileFooter{
     }
     
     /**
    +* Footer for indexed carbon file
    +*/
    +struct FileFooterVersion3{
    --- End diff --
   
    After adding this new struct for V3 format, following is not required in `FileFooter`
    ```8: optional list<BlockletInfo3> blocklet_info_list3;```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r104915516
 
    --- Diff: format/src/main/thrift/carbondata.thrift ---
    @@ -183,11 +183,23 @@ struct FileFooter{
     }
     
     /**
    +* Footer for indexed carbon file
    --- End diff --
   
    Can you change all comment for V3 format, mentioning this struct is for V3 format


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user kumarvishal09 commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r104962541
 
    --- Diff: format/src/main/thrift/carbondata.thrift ---
    @@ -183,11 +183,23 @@ struct FileFooter{
     }
     
     /**
    +* Footer for indexed carbon file
    +*/
    +struct FileFooterVersion3{
    --- End diff --
   
    Ya i will update


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user kumarvishal09 commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r104962645
 
    --- Diff: format/src/main/thrift/carbondata.thrift ---
    @@ -183,11 +183,23 @@ struct FileFooter{
     }
     
     /**
    +* Footer for indexed carbon file
    +*/
    +struct FileFooterVersion3{
    +    1: required i64 num_rows; // Total number of rows in this file
    +    2: required SegmentInfo segment_info; // Segment info (will be same/repeated for all files in this segment)
    +    3: required list<BlockletIndex> blocklet_index_list; // blocklet index of all blocklets in this file
    +    4: optional list<BlockletInfo3> blocklet_info_list3; // Information about blocklets of all columns in this file
    +    5: optional dictionary.ColumnDictionaryChunk dictionary; // blocklet local dictionary
    +}
    +
    +/**
      * Header for appendable carbon file
    --- End diff --
   
    Yes it will support both, appendable and columnar format


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user kumarvishal09 commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r104963110
 
    --- Diff: format/src/main/thrift/carbondata_index.thrift ---
    @@ -41,6 +41,6 @@ struct IndexHeader{
     struct BlockIndex{
       1: required i64 num_rows; // Total number of rows in this file
       2: required string file_name; // Block file name
    -  3: required i64 offset; // Offset of block
    +  3: required i64 offset; // Offset of the footer
    --- End diff --
   
    It's an old code, I m not sure whether changing the variable name will cause compatibility issue or not.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user kumarvishal09 commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r104963505
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/store/writer/v3/CarbonFactDataWriterImplV3.java ---
    @@ -319,11 +322,12 @@ private void writeDataToFile(FileChannel channel, byte[][] dataChunkBytes) {
         // write the header
         try {
           if (fileChannel.size() == 0) {
    -        ColumnarFormatVersion version = CarbonProperties.getInstance().getFormatVersion();
    -        byte[] header = (CarbonCommonConstants.CARBON_DATA_VERSION_HEADER + version).getBytes();
    -        ByteBuffer buffer = ByteBuffer.allocate(header.length);
    -        buffer.put(header);
    -        buffer.rewind();
    +        // below code is to write the file header
    +        byte[] byteArray =
    --- End diff --
   
    ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user kumarvishal09 commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r104963590
 
    --- Diff: format/src/main/thrift/carbondata.thrift ---
    @@ -183,11 +183,23 @@ struct FileFooter{
     }
     
     /**
    +* Footer for indexed carbon file
    +*/
    +struct FileFooterVersion3{
    +    1: required i64 num_rows; // Total number of rows in this file
    +    2: required SegmentInfo segment_info; // Segment info (will be same/repeated for all files in this segment)
    +    3: required list<BlockletIndex> blocklet_index_list; // blocklet index of all blocklets in this file
    +    4: optional list<BlockletInfo3> blocklet_info_list3; // Information about blocklets of all columns in this file
    +    5: optional dictionary.ColumnDictionaryChunk dictionary; // blocklet local dictionary
    +}
    +
    +/**
      * Header for appendable carbon file
      */
     struct FileHeader{
      1: required i32 version; // version used for data compatibility
      2: required list<schema.ColumnSchema> table_columns;  // Description of columns in this file
    + 3: optional bool is_Footer_Present; //  to check whether footer is present or not      
    --- End diff --
   
    MutationType?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user kumarvishal09 commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r104963859
 
    --- Diff: format/src/main/thrift/carbondata.thrift ---
    @@ -183,11 +183,23 @@ struct FileFooter{
     }
     
     /**
    +* Footer for indexed carbon file
    --- End diff --
   
    In reader and writer class?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user kumarvishal09 commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r104971719
 
    --- Diff: format/src/main/thrift/carbondata.thrift ---
    @@ -183,11 +183,23 @@ struct FileFooter{
     }
     
     /**
    +* Footer for indexed carbon file
    --- End diff --
   
    I have updated comments in reader and writer class


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/631
 
    Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1046/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r105051400
 
    --- Diff: format/src/main/thrift/carbondata_index.thrift ---
    @@ -41,6 +41,6 @@ struct IndexHeader{
     struct BlockIndex{
       1: required i64 num_rows; // Total number of rows in this file
       2: required string file_name; // Block file name
    -  3: required i64 offset; // Offset of block
    +  3: required i64 offset; // Offset of the footer
    --- End diff --
   
    Ok, don't change if it, backward compatibility is required


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r105151268
 
    --- Diff: pom.xml ---
    @@ -93,6 +93,7 @@
     
       <modules>
         <module>common</module>
    + <module>format</module>
    --- End diff --
   
    this is not required


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r105151684
 
    --- Diff: format/src/main/thrift/carbondata.thrift ---
    @@ -178,16 +178,27 @@ struct FileFooter{
         5: required list<BlockletIndex> blocklet_index_list; // blocklet index of all blocklets in this file
         6: optional list<BlockletInfo> blocklet_info_list; // Information about blocklets of all columns in this file
         7: optional list<BlockletInfo2> blocklet_info_list2; // Information about blocklets of all columns in this file
    -    8: optional list<BlockletInfo3> blocklet_info_list3; // Information about blocklets of all columns in this file
    -    9: optional dictionary.ColumnDictionaryChunk dictionary; // blocklet local dictionary
    +    8: optional dictionary.ColumnDictionaryChunk dictionary; // blocklet local dictionary
    +}
    +
    +/**
    +* Footer for indexed carbon file
    --- End diff --
   
    I mean, can you mention in this comment that this struct is for V3 carbon file only


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r105151879
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java ---
    @@ -114,17 +116,18 @@ private static FileFooter getFileFooter(List<BlockletInfoColumnar> infoList, int
       }
     
       /**
    -   * It converts list of BlockletInfoColumnar to FileFooter thrift objects
    +   * Below method prepares the file footer object for carbon data file version
    --- End diff --
   
    missed `version 3`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/631
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1059/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user kumarvishal09 commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/631#discussion_r105153484
 
    --- Diff: format/src/main/thrift/carbondata.thrift ---
    @@ -178,16 +178,27 @@ struct FileFooter{
         5: required list<BlockletIndex> blocklet_index_list; // blocklet index of all blocklets in this file
         6: optional list<BlockletInfo> blocklet_info_list; // Information about blocklets of all columns in this file
         7: optional list<BlockletInfo2> blocklet_info_list2; // Information about blocklets of all columns in this file
    -    8: optional list<BlockletInfo3> blocklet_info_list3; // Information about blocklets of all columns in this file
    -    9: optional dictionary.ColumnDictionaryChunk dictionary; // blocklet local dictionary
    +    8: optional dictionary.ColumnDictionaryChunk dictionary; // blocklet local dictionary
    +}
    +
    +/**
    +* Footer for indexed carbon file
    --- End diff --
   
    ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---