Re: The size of the tablestatus file is getting larger, does it impact the performance of reading this file?

Posted by xm_zzc on Mar 15, 2018; 4:49am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/The-size-of-the-tablestatus-file-is-getting-larger-does-it-impact-the-performance-of-reading-this-fi-tp41941p42236.html

Hi Jacky, Raghunandan S:
  Thanks for your reply.
  Currently I am working on PR2045, this pr will automatically delete the
segment lock files when execute method
'SegmentStatusManager.deleteLoadsAndUpdateMetadata', and it will scan
'tablestatus' file to decide which segment lock file need to be deleted.
Ravindra Pesala considers the performance  of reading tablestatus file as
the size of it is getting larger. So I want to know whether it can reduce
the size of tablestatus file.
  According to Raghunandan S's suggestion, I think we can *append* the
invisible segment list to the file called 'tablestatus.history' when execute
command 'CLEAN FILES FOR TABLE' every time, separate  visible and invisible
segments into two files. If later it needs to support listing all
segments(include visible and invisible) list when execute 'SHOW SEGMENTS FOR
TABLE', it just need to read from two files. Is it OK to do so?



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/