Login  Register

Re: [Discussion] Improve the reading/writing performance on the big tablestatus file

Posted by David CaiQiang on Sep 02, 2020; 3:56am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Discussion-Improve-the-reading-writing-performance-on-the-big-tablestatus-file-tp99716p99718.html

add solution 4 to separate the status file by segment status

*solution 4:*   Based on solution 2, support status.inprogress

  1) new tablestatus file format
    {
     "statusFileName":"status-uuid1",
     "inProgressStatusFileName": "status-uuid2.inprogess",
     "updateStatusFileName":"updatestatus-timestamp1",
     "historyStatusFileName":"status.history",
     "segmentMaxId":"1000"
    }

  2) status.inprogess file store the in-progress segment metadata

    Write: at the begin of loading/compaction, add in-progress segment
metadata into status-uuid2.inprogess. at the end, move it to status-uuid1.

    Read: query read status-uuid1 only.  other cases read
status-uuid2.inprogess if needed.



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Best Regards
David Cai