[GitHub] carbondata pull request #1146: [WIP] Change Query related RDD to use TableIn...

[qiuchenjian-2]

GitHub user jackylk opened a pull request:

    https://github.com/apache/carbondata/pull/1146

    [WIP] Change Query related RDD to use TableInfo

    Currently query related RDD is using CarbonTable which is read from file system, which introduces unnecessary file read. We can pass this schema information from driver by serializing TableInfo.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jackylk/incubator-carbondata scanrdd

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1146.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1146
   
----
commit 6597f8ee790c902a6878e09b6bc8dc3915f740e1
Author: jackylk <[hidden email]>
Date:   2017-07-08T09:34:44Z

    change rdd tableinfo

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2957/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/369/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2976/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/387/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2977/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2978/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/388/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/389/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/390/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2979/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2982/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/393/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/407/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1146
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2996/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1146#discussion_r126587482
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/block/AbstractIndex.java ---
    @@ -69,7 +69,7 @@ public SegmentProperties getSegmentProperties() {
         return segmentProperties;
       }
     
    -  /**
    +  /**`
    --- End diff --
   
    please remove this change


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1146#discussion_r126588964
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java ---
    @@ -18,10 +18,14 @@
     package org.apache.carbondata.core.metadata.schema.table;
     
     import java.io.Serializable;
    -import java.util.*;
    +import java.util.ArrayList;
    +import java.util.Collections;
    +import java.util.Comparator;
    +import java.util.HashMap;
    +import java.util.Iterator;
    +import java.util.List;
    +import java.util.Map;
    --- End diff --
   
    use `import java.util.*;`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1146#discussion_r126589927
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableInfo.java ---
    @@ -206,4 +202,68 @@ public void setStorePath(String storePath) {
         }
         return true;
       }
    +
    +  /**
    +   * This method will return the table size. Default table block size will be considered
    +   * in case not specified by the user
    +   */
    +  int getTableBlockSizeInMB() {
    +    String tableBlockSize = null;
    +    // In case of old store there will not be any map for table properties so table properties
    +    // will be null
    +    Map<String, String> tableProperties = getFactTable().getTableProperties();
    +    if (null != tableProperties) {
    +      tableBlockSize = tableProperties.get(CarbonCommonConstants.TABLE_BLOCKSIZE);
    +    }
    +    if (null == tableBlockSize) {
    +      tableBlockSize = CarbonCommonConstants.BLOCK_SIZE_DEFAULT_VAL;
    +      LOGGER.info("Table block size not specified for " + getTableUniqueName()
    +          + ". Therefore considering the default value "
    +          + CarbonCommonConstants.BLOCK_SIZE_DEFAULT_VAL + " MB");
    +    }
    +    return Integer.parseInt(tableBlockSize);
    +  }
    +
    +  @Override
    +  public void write(DataOutput out) throws IOException {
    +    WritableUtil.writeString(out, databaseName);
    --- End diff --
   
    just use `out.writeUTF` for all string writings, no seperate util is required


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1146#discussion_r126590025
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableInfo.java ---
    @@ -206,4 +202,68 @@ public void setStorePath(String storePath) {
         }
         return true;
       }
    +
    +  /**
    +   * This method will return the table size. Default table block size will be considered
    +   * in case not specified by the user
    +   */
    +  int getTableBlockSizeInMB() {
    +    String tableBlockSize = null;
    +    // In case of old store there will not be any map for table properties so table properties
    +    // will be null
    +    Map<String, String> tableProperties = getFactTable().getTableProperties();
    +    if (null != tableProperties) {
    +      tableBlockSize = tableProperties.get(CarbonCommonConstants.TABLE_BLOCKSIZE);
    +    }
    +    if (null == tableBlockSize) {
    +      tableBlockSize = CarbonCommonConstants.BLOCK_SIZE_DEFAULT_VAL;
    +      LOGGER.info("Table block size not specified for " + getTableUniqueName()
    +          + ". Therefore considering the default value "
    +          + CarbonCommonConstants.BLOCK_SIZE_DEFAULT_VAL + " MB");
    +    }
    +    return Integer.parseInt(tableBlockSize);
    +  }
    +
    +  @Override
    +  public void write(DataOutput out) throws IOException {
    +    WritableUtil.writeString(out, databaseName);
    +    WritableUtil.writeString(out, tableUniqueName);
    +    factTable.write(out);
    +    out.writeLong(lastUpdatedTime);
    +    WritableUtil.writeString(out, metaDataFilepath);
    +    WritableUtil.writeString(out, storePath);
    +  }
    +
    +  @Override
    +  public void readFields(DataInput in) throws IOException {
    +    this.databaseName = WritableUtil.readString(in);
    --- End diff --
   
    just use `in.readUTF` for reading string


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1146#discussion_r126590134
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableInfo.java ---
    @@ -206,4 +202,68 @@ public void setStorePath(String storePath) {
         }
         return true;
       }
    +
    +  /**
    +   * This method will return the table size. Default table block size will be considered
    +   * in case not specified by the user
    +   */
    +  int getTableBlockSizeInMB() {
    +    String tableBlockSize = null;
    +    // In case of old store there will not be any map for table properties so table properties
    +    // will be null
    +    Map<String, String> tableProperties = getFactTable().getTableProperties();
    +    if (null != tableProperties) {
    +      tableBlockSize = tableProperties.get(CarbonCommonConstants.TABLE_BLOCKSIZE);
    +    }
    +    if (null == tableBlockSize) {
    +      tableBlockSize = CarbonCommonConstants.BLOCK_SIZE_DEFAULT_VAL;
    +      LOGGER.info("Table block size not specified for " + getTableUniqueName()
    +          + ". Therefore considering the default value "
    +          + CarbonCommonConstants.BLOCK_SIZE_DEFAULT_VAL + " MB");
    +    }
    +    return Integer.parseInt(tableBlockSize);
    +  }
    +
    +  @Override
    +  public void write(DataOutput out) throws IOException {
    +    WritableUtil.writeString(out, databaseName);
    +    WritableUtil.writeString(out, tableUniqueName);
    +    factTable.write(out);
    +    out.writeLong(lastUpdatedTime);
    +    WritableUtil.writeString(out, metaDataFilepath);
    +    WritableUtil.writeString(out, storePath);
    +  }
    +
    +  @Override
    +  public void readFields(DataInput in) throws IOException {
    +    this.databaseName = WritableUtil.readString(in);
    +    this.tableUniqueName = WritableUtil.readString(in);
    +    this.factTable = new TableSchema();
    +    this.factTable.readFields(in);
    +    this.lastUpdatedTime = in.readLong();
    +    this.metaDataFilepath = WritableUtil.readString(in);
    +    this.storePath = WritableUtil.readString(in);
    +  }
    +
    +  public AbsoluteTableIdentifier getOrCreateAbsoluteTableIdentifier() {
    +    if (identifier == null) {
    +      CarbonTableIdentifier carbontableIdentifier =
    +          new CarbonTableIdentifier(databaseName, factTable.getTableName(), factTable.getTableId());
    +      identifier = new AbsoluteTableIdentifier(storePath, carbontableIdentifier);
    +    }
    +    return identifier;
    +  }
    +
    +  public byte[] serielize() throws IOException {
    --- End diff --
   
    Typo, change to `serialize`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---