Posted by
Chin Wei on
Nov 13, 2019; 7:48am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Concurrent-data-loading-issues-tp86108p86256.html
Thanks for your response.
I am running manual compaction, basically the flow are:
1. Multiple (4) threads that running the load data command (each load is
100k rows). I am loading data continuously up to 10,000 csv files.
2. For every 10 load data command, calling the compaction command in another
thread. This is call along with data loading.
I get another error in load data command if I call the 'CLEAN FILES FOR
TABLE' command in another thread along the load data command.
ERROR CarbonLoaderUtil:331 - Entry not found to update
LoadMetadataDetails{loadStatus=Success, loadName='55',
loadStartTime='1573627227495', segmentFile='55_1573627227495.segment'} From
list :: [LoadMetadataDetails{loadStatus=Compacted, loadName='0',
loadStartTime='1573627193399', segmentFile='0_1573627193399.segment'},
LoadMetadataDetails{loadStatus=Success, loadName='0.1',
loadStartTime='1573627205355', segmentFile='0.1_1573627205355.segment'},
LoadMetadataDetails{loadStatus=Success, loadName='10.1',
loadStartTime='1573627205355', segmentFile='10.1_1573627205355.segment'},
LoadMetadataDetails{loadStatus=Success, loadName='20.1',
loadStartTime='1573627216942', segmentFile='20.1_1573627216942.segment'},
LoadMetadataDetails{loadStatus=Success, loadName='50',
loadStartTime='1573627225090', segmentFile='50_1573627225090.segment'},
LoadMetadataDetails{loadStatus=Success, loadName='51',
loadStartTime='1573627225750', segmentFile='51_1573627225750.segment'},
LoadMetadataDetails{loadStatus=Success, loadName='30.1',
loadStartTime='1573627224848', segmentFile='30.1_1573627224848.segment'},
LoadMetadataDetails{loadStatus=Success, loadName='52',
loadStartTime='1573627226311', segmentFile='52_1573627226311.segment'},
LoadMetadataDetails{loadStatus=Success, loadName='53',
loadStartTime='1573627226913', segmentFile='53_1573627226913.segment'},
LoadMetadataDetails{loadStatus=Success, loadName='40.1',
loadStartTime='1573627224848', segmentFile='40.1_1573627224848.segment'},
LoadMetadataDetails{loadStatus=Success, loadName='54',
loadStartTime='1573627227492', segmentFile='54_1573627227492.segment'},
LoadMetadataDetails{loadStatus=Success, loadName='56',
loadStartTime='1573627228051', segmentFile='56_1573627228051.segment'},
LoadMetadataDetails{loadStatus=Insert In Progress, loadName='57',
loadStartTime='1573627228635', segmentFile='null'}]
It seems that the segment 55 is added to tablestatus after CLEAN FILES
command read all the segment from tablestatus, and when CLEAN FILES command
complete, it does not write segment 55 into the tablestatus. Then, when
segment 55 load complete, it is unable to update the tablestatus with that
error.
--
Sent from:
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/