Apache CarbonData Dev Mailing List archive › Apache CarbonData JIRA issues

[jira] [Updated] (CARBONDATA-3904) insert into data got Failed to create directory path /d

classic Classic list List threaded Threaded
♦
♦
 1 message Options
Akash R Nilugal (Jira)
Reply | Threaded
Open this post in threaded view
♦
♦
|

[jira] [Updated] (CARBONDATA-3904) insert into data got Failed to create directory path /d

Akash R Nilugal (Jira)

     [ https://issues.apache.org/jira/browse/CARBONDATA-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XiaoWen updated CARBONDATA-3904:
--------------------------------
    Description:
insert data
{code:java}
spark.sql("INSERT OVERWRITE TABLE ods.test_table SELECT * FROM ods.socol_cmdinfo")
{code}
 check logs from spark application on yarn

$ yarn logs -applicationId application_1592787941917_4116

found a lot this error messages
{code:java}
20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:12:45 ERROR FileFactory:  Failed to create directory path /d
{code}
{code:java}
core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
{code}
{code:java}
public static void createDirectoryAndSetPermission(String directoryPath, FsPermission permission)
      throws IOException {
    FileFactory.FileType fileType = FileFactory.getFileType(directoryPath);
    switch (fileType) {
      case S3:
      case HDFS:
      case ALLUXIO:
      case VIEWFS:
      case CUSTOM:
      case HDFS_LOCAL:
        try {
          Path path = new Path(directoryPath);
          FileSystem fs = path.getFileSystem(getConfiguration());
          if (!fs.exists(path)) {
            fs.mkdirs(path);
            fs.setPermission(path, permission);
          }
        } catch (IOException e) {
          LOGGER.error("Exception occurred : " + e.getMessage(), e);
          throw e;
        }
        return;
      case LOCAL:
      default:
        directoryPath = FileFactory.getUpdatedFilePath(directoryPath);
        File file = new File(directoryPath);
        if (!file.mkdirs()) {
          LOGGER.error(" Failed to create directory path " + directoryPath);
        }    }
  }
{code}
 

I output the variable directoryPath and fileType
{code:java}
if (!file.mkdirs()) {
          //  check variables
          LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");
          LOGGER.error(" Failed to create directory path " + directoryPath);
        }
{code}
add line 

LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");

got echo on yarn logs

2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]

....

 

Why fileType is LOCAL?

I have never set this value: directoryPath = [/d]?

But the data is inserted normally and can be queried normally.

  was:
insert data

 
{code:java}
spark.sql("INSERT OVERWRITE TABLE ods.test_table SELECT * FROM ods.socol_cmdinfo")
{code}
 

 check logs from spark application on yarn

$ yarn logs -applicationId application_1592787941917_4116

found a lot this error messages
{code:java}
20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:12:45 ERROR FileFactory:  Failed to create directory path /d
{code}
 
{code:java}
core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
{code}
{code:java}
public static void createDirectoryAndSetPermission(String directoryPath, FsPermission permission)
      throws IOException {
    FileFactory.FileType fileType = FileFactory.getFileType(directoryPath);
    switch (fileType) {
      case S3:
      case HDFS:
      case ALLUXIO:
      case VIEWFS:
      case CUSTOM:
      case HDFS_LOCAL:
        try {
          Path path = new Path(directoryPath);
          FileSystem fs = path.getFileSystem(getConfiguration());
          if (!fs.exists(path)) {
            fs.mkdirs(path);
            fs.setPermission(path, permission);
          }
        } catch (IOException e) {
          LOGGER.error("Exception occurred : " + e.getMessage(), e);
          throw e;
        }
        return;
      case LOCAL:
      default:
        directoryPath = FileFactory.getUpdatedFilePath(directoryPath);
        File file = new File(directoryPath);
        if (!file.mkdirs()) {
          LOGGER.error(" Failed to create directory path " + directoryPath);
        }    }
  }
{code}
x

 
{code:java}
if (!file.mkdirs()) {
          //  check variables
          LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");
          LOGGER.error(" Failed to create directory path " + directoryPath);
        }
{code}
add line 

LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");

got echo on yarn logs

2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]

....

 

Why fileType is LOCAL?

I have never set this value: directoryPath = [/d]?

But the data is inserted normally and can be queried normally

 

 

 

 

x


> insert into data got Failed to create directory path /d
> -------------------------------------------------------
>
>                 Key: CARBONDATA-3904
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-3904
>             Project: CarbonData
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 2.0.0
>         Environment: spark-2.4.5
> hadoop 2.7.3
> carbondata2.0.1
>            Reporter: XiaoWen
>            Priority: Minor
>
> insert data
> {code:java}
> spark.sql("INSERT OVERWRITE TABLE ods.test_table SELECT * FROM ods.socol_cmdinfo")
> {code}
>  check logs from spark application on yarn
> $ yarn logs -applicationId application_1592787941917_4116
> found a lot this error messages
> {code:java}
> 20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
> 20/07/15 17:12:45 ERROR FileFactory:  Failed to create directory path /d
> {code}
> {code:java}
> core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
> {code}
> {code:java}
> public static void createDirectoryAndSetPermission(String directoryPath, FsPermission permission)
>       throws IOException {
>     FileFactory.FileType fileType = FileFactory.getFileType(directoryPath);
>     switch (fileType) {
>       case S3:
>       case HDFS:
>       case ALLUXIO:
>       case VIEWFS:
>       case CUSTOM:
>       case HDFS_LOCAL:
>         try {
>           Path path = new Path(directoryPath);
>           FileSystem fs = path.getFileSystem(getConfiguration());
>           if (!fs.exists(path)) {
>             fs.mkdirs(path);
>             fs.setPermission(path, permission);
>           }
>         } catch (IOException e) {
>           LOGGER.error("Exception occurred : " + e.getMessage(), e);
>           throw e;
>         }
>         return;
>       case LOCAL:
>       default:
>         directoryPath = FileFactory.getUpdatedFilePath(directoryPath);
>         File file = new File(directoryPath);
>         if (!file.mkdirs()) {
>           LOGGER.error(" Failed to create directory path " + directoryPath);
>         }    }
>   }
> {code}
>  
> I output the variable directoryPath and fileType
> {code:java}
> if (!file.mkdirs()) {
>           //  check variables
>           LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");
>           LOGGER.error(" Failed to create directory path " + directoryPath);
>         }
> {code}
> add line 
> LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");
> got echo on yarn logs
> 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
>  2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
> ....
>  
> Why fileType is LOCAL?
> I have never set this value: directoryPath = [/d]?
> But the data is inserted normally and can be queried normally.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Free forum by Nabble Edit this page