Re: The size of the tablestatus file is getting larger, does it impact the performance of reading this file?

Posted by Jacky Li on Mar 15, 2018; 8:05am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/The-size-of-the-tablestatus-file-is-getting-larger-does-it-impact-the-performance-of-reading-this-fi-tp41941p42283.html

Hi,

I think this approach (maitaining a history tablestatus file) is good.
Xm_zzc, please continue with this approach if you want to work on it.

Regards,
Jacky

> 在 2018年3月15日，下午1:47，manish gupta <[hidden email]> 写道：
>
> I think maintaining a tablestatus backlog file is a good idea. This will
> also help us in quick filtering of valid segments as the number of segments
> increase during queries execution which involve reading of table status
> file.
>
> Show segment DDL can read both the files to display the output.
>
> Regards
> Manish Gupta
>
> On Thu, 15 Mar 2018 at 10:19 AM, xm_zzc <[hidden email]> wrote:
>
>> Hi Jacky, Raghunandan S:
>>  Thanks for your reply.
>>  Currently I am working on PR2045, this pr will automatically delete the
>> segment lock files when execute method
>> 'SegmentStatusManager.deleteLoadsAndUpdateMetadata', and it will scan
>> 'tablestatus' file to decide which segment lock file need to be deleted.
>> Ravindra Pesala considers the performance  of reading tablestatus file as
>> the size of it is getting larger. So I want to know whether it can reduce
>> the size of tablestatus file.
>>  According to Raghunandan S's suggestion, I think we can *append* the
>> invisible segment list to the file called 'tablestatus.history' when
>> execute
>> command 'CLEAN FILES FOR TABLE' every time, separate  visible and invisible
>> segments into two files. If later it needs to support listing all
>> segments(include visible and invisible) list when execute 'SHOW SEGMENTS
>> FOR
>> TABLE', it just need to read from two files. Is it OK to do so?
>>
>>
>>
>> --
>> Sent from:
>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>>