niuge01 opened a new pull request #3570: [HOTFIX] Fix FileNotFound error when concurrent loading
URL:
https://github.com/apache/carbondata/pull/3570
### Why is this PR needed?
When multiple threads use SDK write data, they will transfer CarbonLoadModel parameter with the same Configuration object, the following process may use an CarbonLoadModel which set by other thread, this will lead to a series of problems, such as:
2020-01-09 14:42:47 ERROR CarbonFactDataWriterImplV3:390 - Problem while writing the index file
org.apache.carbondata.core.datastore.exception.CarbonDataWriterException: Problem while copying file from local store to carbon store
at org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2772)
at org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2721)
at org.apache.carbondata.processing.store.writer.AbstractFactDataWriter.commitCurrentFile(AbstractFactDataWriter.java:277)
at org.apache.carbondata.processing.store.writer.v3.CarbonFactDataWriterImplV3.closeWriter(CarbonFactDataWriterImplV3.java:387)
at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.closeHandler(CarbonFactDataHandlerColumnar.java:508)
at org.apache.carbondata.processing.loading.steps.CarbonRowDataWriterProcessorStepImpl.processingComplete(CarbonRowDataWriterProcessorStepImpl.java:233)
at org.apache.carbondata.processing.loading.steps.CarbonRowDataWriterProcessorStepImpl.finish(CarbonRowDataWriterProcessorStepImpl.java:211)
at org.apache.carbondata.processing.loading.steps.CarbonRowDataWriterProcessorStepImpl.doExecute(CarbonRowDataWriterProcessorStepImpl.java:175)
at org.apache.carbondata.processing.loading.steps.CarbonRowDataWriterProcessorStepImpl.execute(CarbonRowDataWriterProcessorStepImpl.java:129)
at org.apache.carbondata.processing.loading.DataLoadExecutor.execute(DataLoadExecutor.java:52)
at org.apache.carbondata.hadoop.api.CarbonTableOutputFormat$1.run(CarbonTableOutputFormat.java:278)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: D:\Workspace\Carbon-Master\integration\flink\target\test-classes\data\temp\8df45d4dc38449c69147083cdbe79e4d\part-0-156_batchno0-0-null-1578552167465.carbondata (系统找不到指定的路径。)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:101)
at org.apache.carbondata.core.datastore.filesystem.LocalCarbonFile.getDataOutputStream(LocalCarbonFile.java:371)
at org.apache.carbondata.core.datastore.filesystem.LocalCarbonFile.getDataOutputStream(LocalCarbonFile.java:365)
at org.apache.carbondata.core.datastore.impl.FileFactory.getDataOutputStream(FileFactory.java:231)
at org.apache.carbondata.core.util.CarbonUtil.copyLocalFileToCarbonStore(CarbonUtil.java:2799)
at org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2756)
... 15 more
### What changes were proposed in this PR?
Different thread use different configuration object.
### Does this PR introduce any user interface change?
- No
### Is any new testcase added?
- No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[hidden email]
With regards,
Apache Git Services
Locked
