[GitHub] carbondata pull request #2131: [WIP] Support unmanaged carbon table read and...

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3568/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3569/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4792/



---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2131#discussion_r179032115
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonCountStar.scala ---
    @@ -76,6 +76,11 @@ case class CarbonCountStar(
         SparkHadoopUtil.get.addCredentials(jobConf)
         val job = new Job(jobConf)
         FileInputFormat.addInputPath(job, new Path(absoluteTableIdentifier.getTablePath))
    +    if (carbonTable.getTableInfo.isUnManagedTable) {
    +      CarbonInputFormat.setCarbonUnmanagedTable(job.getConfiguration, "true")
    --- End diff --
   
    setter change it to boolean


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4809/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3585/



---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2131#discussion_r179098290
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonCountStar.scala ---
    @@ -76,6 +76,11 @@ case class CarbonCountStar(
         SparkHadoopUtil.get.addCredentials(jobConf)
         val job = new Job(jobConf)
         FileInputFormat.addInputPath(job, new Path(absoluteTableIdentifier.getTablePath))
    +    if (carbonTable.getTableInfo.isUnManagedTable) {
    +      CarbonInputFormat.setCarbonUnmanagedTable(job.getConfiguration, "true")
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Retest this please


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2131#discussion_r179085175
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java ---
    @@ -152,15 +167,23 @@ private CarbonTable buildCarbonTable() {
               new StructField(field.getFieldName(), field.getDataType()),
               sortColumnsList.contains(field.getFieldName()));
         }
    -    String tableName = "_tempTable";
    -    String dbName = "_tempDB";
    +    String tableName;
    --- End diff --
   
    Not required to add tablename here, in this class, keep it same as previous


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2131#discussion_r179088904
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/model/CarbonLoadModel.java ---
    @@ -47,6 +47,13 @@
     
       private String tablePath;
     
    +  /*
    +     This points if the carbonTable is a Unmanaged Table or not.
    +     The path will be pointed by the tablePath. And there will be
    +     no Metadata folder present for the unmanaged Table.
    +   */
    +  private boolean carbonUnmanagedTable;
    --- End diff --
   
    already class name has carbon word, so not required to repeat, this again. better use "unmanagedTable"


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4820/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3597/



---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2131#discussion_r179148873
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/model/CarbonLoadModel.java ---
    @@ -47,6 +47,13 @@
     
       private String tablePath;
     
    +  /*
    +     This points if the carbonTable is a Unmanaged Table or not.
    +     The path will be pointed by the tablePath. And there will be
    +     no Metadata folder present for the unmanaged Table.
    +   */
    +  private boolean carbonUnmanagedTable;
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2131#discussion_r179150325
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java ---
    @@ -128,12 +134,18 @@ protected CarbonTable getOrCreateCarbonTable(Configuration configuration) throws
       @Override
       public List<InputSplit> getSplits(JobContext job) throws IOException {
         AbsoluteTableIdentifier identifier = getAbsoluteTableIdentifier(job.getConfiguration());
    -    LoadMetadataDetails[] loadMetadataDetails = SegmentStatusManager
    -        .readTableStatusFile(CarbonTablePath.getTableStatusFilePath(identifier.getTablePath()));
    +
         CarbonTable carbonTable = getOrCreateCarbonTable(job.getConfiguration());
         if (null == carbonTable) {
           throw new IOException("Missing/Corrupt schema file for table.");
         }
    +    this.readCommittedScope = getReadCommitted(job, identifier);
    --- End diff --
   
    When empty data.. select should give empty records


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2131#discussion_r179154952
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
    @@ -2910,10 +2914,16 @@ private static long getMaxOfBlockAndFileSize(long blockSize, long fileSize) {
        * @return
        */
       public static String getBlockId(AbsoluteTableIdentifier identifier, String filePath,
    -      String segmentId) {
    +      String segmentId, boolean isUnmangedTable) {
         String blockId;
         String blockName = filePath.substring(filePath.lastIndexOf("/") + 1, filePath.length());
         String tablePath = identifier.getTablePath();
    +
    --- End diff --
   
    this block id should be commonly constructed for unmanaged and managed case. As unmanged partition is also possible


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4294/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4295/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4827/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2131
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3604/



---