[jira] [Commented] (CARBONDATA-3904) insert into data got Failed to create directory path /d

[Akash R Nilugal (Jira)]

    [ https://issues.apache.org/jira/browse/CARBONDATA-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17158972#comment-17158972 ]

Kunal Kapoor commented on CARBONDATA-3904:
------------------------------------------

What is the warehouse location? HDFS/S3?

> insert into data got Failed to create directory path /d
> -------------------------------------------------------
>
>                 Key: CARBONDATA-3904
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-3904
>             Project: CarbonData
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 2.0.0
>         Environment: spark-2.4.5
> hadoop 2.7.3
> carbondata2.0.1
>            Reporter: XiaoWen
>            Priority: Minor
>
> insert data
> {code:java}
> spark.sql("INSERT OVERWRITE TABLE ods.test_table SELECT * FROM ods.socol_cmdinfo")
> {code}
>  check logs from spark application on yarn
> $ yarn logs -applicationId application_1592787941917_4116
> found a lot this error messages
> {code:java}
> 20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:12:45 ERROR FileFactory:  Failed to create directory path /d
> {code}
> {code:java}
> core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
> {code}
> {code:java}
> public static void createDirectoryAndSetPermission(String directoryPath, FsPermission permission)
>       throws IOException {
>     FileFactory.FileType fileType = FileFactory.getFileType(directoryPath);
>     switch (fileType) {
>       case S3:
>       case HDFS:
>       case ALLUXIO:
>       case VIEWFS:
>       case CUSTOM:
>       case HDFS_LOCAL:
>         try {
>           Path path = new Path(directoryPath);
>           FileSystem fs = path.getFileSystem(getConfiguration());
>           if (!fs.exists(path)) {
>             fs.mkdirs(path);
>             fs.setPermission(path, permission);
>           }
>         } catch (IOException e) {
>           LOGGER.error("Exception occurred : " + e.getMessage(), e);
>           throw e;
>         }
>         return;
>       case LOCAL:
>       default:
>         directoryPath = FileFactory.getUpdatedFilePath(directoryPath);
>         File file = new File(directoryPath);
>         if (!file.mkdirs()) {
>           LOGGER.error(" Failed to create directory path " + directoryPath);
>         }    }
>   }
> {code}
>  
> I output the variable directoryPath and fileType
> {code:java}
> if (!file.mkdirs()) {
>           //  check variables
>           LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");
>           LOGGER.error(" Failed to create directory path " + directoryPath);
>         }
> {code}
> add line 
> LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");
> got echo on yarn logs
> 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
> ....
>  
> Why fileType is LOCAL?
> I have never set this value: directoryPath = [/d]?
> But the data is inserted normally and can be queried normally.

--
This message was sent by Atlassian Jira
(v8.3.4#803005)