[GitHub] carbondata pull request #2161: Adding Alluxio Supporty

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5155/



---

[qiuchenjian-2]

Github user chandrasaripaka commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    @xubo2545 looks like there is a permission issue for locating the file to build for my I'd. Actually the build is passing.


---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3945/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4028/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5229/



---

[qiuchenjian-2]

Github user chandrasaripaka commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    @xubo245 how do we merge this


---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    it need Committer review and merge this.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4221/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5388/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4526/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5407/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4240/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4545/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2161#discussion_r185777526
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AlluxioCarbonFile.java ---
    @@ -105,18 +103,21 @@ public CarbonFile getParentFile() {
         return null == parent ? null : new AlluxioCarbonFile(parent);
       }
     
    +  /**
    +   * CARBON-2218 Adopting to {@link FileSystem} , in accordance
    +   * with the AlluxioFileSystem Implementation.
    +   * Implicit expection of the AlluxioFileSystem when using this method.
    +   * If success, returns true else false when an Exception is raised.
    +   * @param changetoName
    +   * @return
    +   */
       @Override
       public boolean renameForce(String changetoName) {
         FileSystem fs;
         try {
           fs = fileStatus.getPath().getFileSystem(FileFactory.getConfiguration());
    -      if (fs instanceof DistributedFileSystem) {
    -        ((DistributedFileSystem) fs).rename(fileStatus.getPath(), new Path(changetoName),
    -            org.apache.hadoop.fs.Options.Rename.OVERWRITE);
    -        return true;
    -      } else {
    -        return false;
    -      }
    +      fs.delete(new Path(changetoName), true);
    --- End diff --
   
    It's a dangerous operation as we cannot get back the file once it is removed. So failure cases may lead to system inconsistent.  Since Alluxio does not support rename we should try to bring the framework where we should not use renameForce. We will work on it.


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2161#discussion_r185778044
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java ---
    @@ -365,10 +366,11 @@ public static boolean createNewLockFile(String filePath, FileType fileType) thro
       public static String getUpdatedFilePath(String filePath, FileType fileType) {
         switch (fileType) {
           case HDFS:
    -      case ALLUXIO:
           case VIEWFS:
           case S3:
             return filePath;
    +      case ALLUXIO:
    +        return StringUtils.containsAny(filePath,"alluxio")?filePath: "alluxio://"+filePath;
    --- End diff --
   
    Why is this special handling for alluxio?


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6535/



---

[qiuchenjian-2]

Github user chandrasaripaka commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2161#discussion_r200064110
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AlluxioCarbonFile.java ---
    @@ -105,18 +103,21 @@ public CarbonFile getParentFile() {
         return null == parent ? null : new AlluxioCarbonFile(parent);
       }
     
    +  /**
    +   * CARBON-2218 Adopting to {@link FileSystem} , in accordance
    +   * with the AlluxioFileSystem Implementation.
    +   * Implicit expection of the AlluxioFileSystem when using this method.
    +   * If success, returns true else false when an Exception is raised.
    +   * @param changetoName
    +   * @return
    +   */
       @Override
       public boolean renameForce(String changetoName) {
         FileSystem fs;
         try {
           fs = fileStatus.getPath().getFileSystem(FileFactory.getConfiguration());
    -      if (fs instanceof DistributedFileSystem) {
    -        ((DistributedFileSystem) fs).rename(fileStatus.getPath(), new Path(changetoName),
    -            org.apache.hadoop.fs.Options.Rename.OVERWRITE);
    -        return true;
    -      } else {
    -        return false;
    -      }
    +      fs.delete(new Path(changetoName), true);
    --- End diff --
   
    Any update on this. Can you suggest on a alternative way.


---

[qiuchenjian-2]

Github user chandrasaripaka commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2161#discussion_r200066306
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java ---
    @@ -365,10 +366,11 @@ public static boolean createNewLockFile(String filePath, FileType fileType) thro
       public static String getUpdatedFilePath(String filePath, FileType fileType) {
         switch (fileType) {
           case HDFS:
    -      case ALLUXIO:
           case VIEWFS:
           case S3:
             return filePath;
    +      case ALLUXIO:
    +        return StringUtils.containsAny(filePath,"alluxio")?filePath: "alluxio://"+filePath;
    --- End diff --
   
    Alluxio does the path translation in the FileSystem , that s why we have this issue. https://github.com/Alluxio/alluxio/blob/master/core/client/hdfs/src/main/java/alluxio/hadoop/AbstractFileSystem.java


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2161
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7116/



---