Re: The size of the tablestatus file is getting larger, does it impact the performance of reading this file?

Posted by manishgupta88 on Mar 15, 2018; 5:47am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/The-size-of-the-tablestatus-file-is-getting-larger-does-it-impact-the-performance-of-reading-this-fi-tp41941p42246.html

I think maintaining a tablestatus backlog file is a good idea. This will
also help us in quick filtering of valid segments as the number of segments
increase during queries execution which involve reading of table status
file.

Show segment DDL can read both the files to display the output.

Regards
Manish Gupta

On Thu, 15 Mar 2018 at 10:19 AM, xm_zzc <[hidden email]> wrote:

> Hi Jacky, Raghunandan S:
>   Thanks for your reply.
>   Currently I am working on PR2045, this pr will automatically delete the
> segment lock files when execute method
> 'SegmentStatusManager.deleteLoadsAndUpdateMetadata', and it will scan
> 'tablestatus' file to decide which segment lock file need to be deleted.
> Ravindra Pesala considers the performance  of reading tablestatus file as
> the size of it is getting larger. So I want to know whether it can reduce
> the size of tablestatus file.
>   According to Raghunandan S's suggestion, I think we can *append* the
> invisible segment list to the file called 'tablestatus.history' when
> execute
> command 'CLEAN FILES FOR TABLE' every time, separate  visible and invisible
> segments into two files. If later it needs to support listing all
> segments(include visible and invisible) list when execute 'SHOW SEGMENTS
> FOR
> TABLE', it just need to read from two files. Is it OK to do so?
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>