[GitHub] carbondata pull request #1239: [CARBONDATA-1338] add tableInfo to CarbonHive...

[qiuchenjian-2]

GitHub user cenyuhai opened a pull request:

    https://github.com/apache/carbondata/pull/1239

    [CARBONDATA-1338] add tableInfo to CarbonHiveInputSplit and no need to get schema from file

    add tableInfo to CarbonHiveInputSplit and no need to get schema from file in map process.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cenyuhai/incubator-carbondata CARBONDATA-1338

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1239.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1239
   
----
commit 29b46b64096829dc0260bb9d562ec7b8c8e329dc
Author: cenyuhai <[hidden email]>
Date:   2017-08-06T05:47:56Z

    add tableInfo to CarbonHiveInputSplit and no need to get schema from file

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    SDV Build Success with Spark 2.1, Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/123/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/796/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3394/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3396/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/798/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    SDV Build Success with Spark 2.1, Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/125/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user cenyuhai commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3400/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/802/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    SDV Build Success with Spark 2.1, Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/129/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1239#discussion_r132685712
 
    --- Diff: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java ---
    @@ -84,47 +92,66 @@
        * @param configuration
        * @throws IOException
        */
    -  private static void populateCarbonTable(Configuration configuration, String paths)
    +  private CarbonTable populateCarbonTable(Configuration configuration)
           throws IOException {
    -    String dirs = configuration.get(INPUT_DIR, "");
    -    String[] inputPaths = StringUtils.split(dirs);
    -    String validInputPath = null;
    +    TableInfo tableInfo = getTableInfo(configuration);
    +    CarbonTable carbonTable = null;
    +    if (tableInfo != null) {
    +      carbonTable = CarbonTable.buildFromTableInfo(tableInfo);
    +      CarbonMetadata.getInstance().addCarbonTable(carbonTable);
    +      return carbonTable;
    +    }
    +    String inputDir = configuration.get(INPUT_DIR, "");
    +    String[] inputPaths = StringUtils.split(inputDir);
         if (inputPaths.length == 0) {
           throw new InvalidPathException("No input paths specified in job");
    -    } else {
    -      if (paths != null) {
    -        for (String inputPath : inputPaths) {
    -          if (paths.startsWith(inputPath)) {
    -            validInputPath = inputPath;
    -            break;
    -          }
    -        }
    -      }
         }
    +    Arrays.sort(inputPaths);
    --- End diff --
   
    why a sort is needed here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1239#discussion_r132685377
 
    --- Diff: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java ---
    @@ -84,47 +92,66 @@
        * @param configuration
        * @throws IOException
        */
    -  private static void populateCarbonTable(Configuration configuration, String paths)
    +  private CarbonTable populateCarbonTable(Configuration configuration)
           throws IOException {
    -    String dirs = configuration.get(INPUT_DIR, "");
    -    String[] inputPaths = StringUtils.split(dirs);
    -    String validInputPath = null;
    +    TableInfo tableInfo = getTableInfo(configuration);
    +    CarbonTable carbonTable = null;
    +    if (tableInfo != null) {
    +      carbonTable = CarbonTable.buildFromTableInfo(tableInfo);
    +      CarbonMetadata.getInstance().addCarbonTable(carbonTable);
    +      return carbonTable;
    +    }
    +    String inputDir = configuration.get(INPUT_DIR, "");
    +    String[] inputPaths = StringUtils.split(inputDir);
         if (inputPaths.length == 0) {
           throw new InvalidPathException("No input paths specified in job");
    -    } else {
    -      if (paths != null) {
    -        for (String inputPath : inputPaths) {
    -          if (paths.startsWith(inputPath)) {
    -            validInputPath = inputPath;
    -            break;
    -          }
    -        }
    -      }
         }
    +    Arrays.sort(inputPaths);
    +    String tablePath = inputPaths[0].replace("file:", "");
    --- End diff --
   
    Is this a mistake?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user cenyuhai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1239#discussion_r132691664
 
    --- Diff: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java ---
    @@ -84,47 +92,66 @@
        * @param configuration
        * @throws IOException
        */
    -  private static void populateCarbonTable(Configuration configuration, String paths)
    +  private CarbonTable populateCarbonTable(Configuration configuration)
           throws IOException {
    -    String dirs = configuration.get(INPUT_DIR, "");
    -    String[] inputPaths = StringUtils.split(dirs);
    -    String validInputPath = null;
    +    TableInfo tableInfo = getTableInfo(configuration);
    +    CarbonTable carbonTable = null;
    +    if (tableInfo != null) {
    +      carbonTable = CarbonTable.buildFromTableInfo(tableInfo);
    +      CarbonMetadata.getInstance().addCarbonTable(carbonTable);
    +      return carbonTable;
    +    }
    +    String inputDir = configuration.get(INPUT_DIR, "");
    +    String[] inputPaths = StringUtils.split(inputDir);
         if (inputPaths.length == 0) {
           throw new InvalidPathException("No input paths specified in job");
    -    } else {
    -      if (paths != null) {
    -        for (String inputPath : inputPaths) {
    -          if (paths.startsWith(inputPath)) {
    -            validInputPath = inputPath;
    -            break;
    -          }
    -        }
    -      }
         }
    +    Arrays.sort(inputPaths);
    --- End diff --
   
    because there are serveral paths, the sortest path are the table path, so I sort all and then get the first one


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user cenyuhai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1239#discussion_r132695250
 
    --- Diff: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java ---
    @@ -84,47 +92,66 @@
        * @param configuration
        * @throws IOException
        */
    -  private static void populateCarbonTable(Configuration configuration, String paths)
    +  private CarbonTable populateCarbonTable(Configuration configuration)
           throws IOException {
    -    String dirs = configuration.get(INPUT_DIR, "");
    -    String[] inputPaths = StringUtils.split(dirs);
    -    String validInputPath = null;
    +    TableInfo tableInfo = getTableInfo(configuration);
    +    CarbonTable carbonTable = null;
    +    if (tableInfo != null) {
    +      carbonTable = CarbonTable.buildFromTableInfo(tableInfo);
    +      CarbonMetadata.getInstance().addCarbonTable(carbonTable);
    +      return carbonTable;
    +    }
    +    String inputDir = configuration.get(INPUT_DIR, "");
    +    String[] inputPaths = StringUtils.split(inputDir);
         if (inputPaths.length == 0) {
           throw new InvalidPathException("No input paths specified in job");
    -    } else {
    -      if (paths != null) {
    -        for (String inputPath : inputPaths) {
    -          if (paths.startsWith(inputPath)) {
    -            validInputPath = inputPath;
    -            break;
    -          }
    -        }
    -      }
         }
    +    Arrays.sort(inputPaths);
    +    String tablePath = inputPaths[0].replace("file:", "");
    --- End diff --
   
    because of another issue, my pr is conflict with https://github.com/apache/carbondata/pull/1231


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user cenyuhai commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    retest please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    SDV Build Failed with Spark 2.1, Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/245/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user anubhav100 commented on the issue:

    https://github.com/apache/carbondata/pull/1239
 
    @cenyuhai this issue is resolved in latest master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---