[jira] [Created] (CARBONDATA-3904) insert into data got Failed to create directory path /d

[Akash R Nilugal (Jira)]

XiaoWen created CARBONDATA-3904:
-----------------------------------

             Summary: insert into data got Failed to create directory path /d
                 Key: CARBONDATA-3904
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-3904
             Project: CarbonData
          Issue Type: Improvement
          Components: core
    Affects Versions: 2.0.0
         Environment: spark-2.4.5
hadoop 2.7.3
carbondata2.0.1
            Reporter: XiaoWen


insert data

 
{code:java}
spark.sql("INSERT OVERWRITE TABLE ods.test_table SELECT * FROM ods.socol_cmdinfo")
{code}
 

 check logs from spark application on yarn

$ yarn logs -applicationId application_1592787941917_4116

found a lot this error messages
{code:java}
20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:12:45 ERROR FileFactory:  Failed to create directory path /d
{code}
 
{code:java}
core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
{code}
{code:java}
public static void createDirectoryAndSetPermission(String directoryPath, FsPermission permission)
      throws IOException {
    FileFactory.FileType fileType = FileFactory.getFileType(directoryPath);
    switch (fileType) {
      case S3:
      case HDFS:
      case ALLUXIO:
      case VIEWFS:
      case CUSTOM:
      case HDFS_LOCAL:
        try {
          Path path = new Path(directoryPath);
          FileSystem fs = path.getFileSystem(getConfiguration());
          if (!fs.exists(path)) {
            fs.mkdirs(path);
            fs.setPermission(path, permission);
          }
        } catch (IOException e) {
          LOGGER.error("Exception occurred : " + e.getMessage(), e);
          throw e;
        }
        return;
      case LOCAL:
      default:
        directoryPath = FileFactory.getUpdatedFilePath(directoryPath);
        File file = new File(directoryPath);
        if (!file.mkdirs()) {
          LOGGER.error(" Failed to create directory path " + directoryPath);
        }    }
  }
{code}
x

 
{code:java}
if (!file.mkdirs()) {
          //  check variables
          LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");
          LOGGER.error(" Failed to create directory path " + directoryPath);
        }
{code}
add line 

LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");

got echo on yarn logs

2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]

....

 

Why fileType is LOCAL?

I have never set this value: directoryPath = [/d]?

But the data is inserted normally and can be queried normally

 

 

 

 

x



--
This message was sent by Atlassian Jira
(v8.3.4#803005)