[GitHub] carbondata pull request #2131: [WIP] Support unmanaged carbon table read and...

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4300/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3606/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4829/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4301/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    retest this please


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2131#discussion_r179360469
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
    @@ -2910,10 +2914,16 @@ private static long getMaxOfBlockAndFileSize(long blockSize, long fileSize) {
        * @return
        */
       public static String getBlockId(AbsoluteTableIdentifier identifier, String filePath,
    -      String segmentId) {
    +      String segmentId, boolean isUnmangedTable) {
         String blockId;
         String blockName = filePath.substring(filePath.lastIndexOf("/") + 1, filePath.length());
         String tablePath = identifier.getTablePath();
    +
    --- End diff --
   
    OK. Hanlded.


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2131#discussion_r179360518
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java ---
    @@ -128,12 +134,18 @@ protected CarbonTable getOrCreateCarbonTable(Configuration configuration) throws
       @Override
       public List<InputSplit> getSplits(JobContext job) throws IOException {
         AbsoluteTableIdentifier identifier = getAbsoluteTableIdentifier(job.getConfiguration());
    -    LoadMetadataDetails[] loadMetadataDetails = SegmentStatusManager
    -        .readTableStatusFile(CarbonTablePath.getTableStatusFilePath(identifier.getTablePath()));
    +
         CarbonTable carbonTable = getOrCreateCarbonTable(job.getConfiguration());
         if (null == carbonTable) {
           throw new IOException("Missing/Corrupt schema file for table.");
         }
    +    this.readCommittedScope = getReadCommitted(job, identifier);
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2131#discussion_r179361367
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/TableStatusReadCommittedScope.java ---
    @@ -0,0 +1,76 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.carbondata.core.readcommitter;
    +
    +import java.io.IOException;
    +import java.util.Map;
    +
    +import org.apache.carbondata.common.annotations.InterfaceAudience;
    +import org.apache.carbondata.common.annotations.InterfaceStability;
    +import org.apache.carbondata.core.datamap.Segment;
    +import org.apache.carbondata.core.indexstore.blockletindex.SegmentIndexFileStore;
    +import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
    +import org.apache.carbondata.core.metadata.SegmentFileStore;
    +import org.apache.carbondata.core.statusmanager.LoadMetadataDetails;
    +import org.apache.carbondata.core.statusmanager.SegmentStatusManager;
    +import org.apache.carbondata.core.util.path.CarbonTablePath;
    +
    +/**
    + * ReadCommittedScope for the managed carbon table
    + */
    [e-mail]
    [e-mail]
    +public class TableStatusReadCommittedScope implements ReadCommittedScope {
    +  private LoadMetadataDetails[] loadMetadataDetails;
    +  private AbsoluteTableIdentifier identifier;
    +
    +  public TableStatusReadCommittedScope(AbsoluteTableIdentifier identifier) {
    +    this.identifier = identifier;
    +    takeCarbonIndexFileSnapShot();
    +  }
    +
    +  @Override public LoadMetadataDetails[] getSegmentList() throws IOException {
    +    try {
    +      this.loadMetadataDetails = SegmentStatusManager
    +          .readTableStatusFile(CarbonTablePath.getTableStatusFilePath(identifier.getTablePath()));
    +      return this.loadMetadataDetails;
    +    } catch (IOException ex) {
    +      this.loadMetadataDetails = new LoadMetadataDetails[0];
    --- End diff --
   
    should throw exception here, it is not valid behaviour to return empty list


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3607/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4830/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4832/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3609/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3611/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4304/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4837/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3614/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4306/



---