Hi Community!
When load/insert command is triggered in the scenario where the main table
has one or more SI tables, after loading the new segment in the main table
and all the SI tables there is a SILoadEventListenerForFailedSegments which
compares the segments in main table and SI table. In case of any mismatch or
missing segments in any of the SI tables, the listener fires a load on the
missing segments in the SI table. The load/insert command on the main table
will be finished only after all the missing segments in all the SI tables
have been loaded again.
Consider a scenario where the SI table has 10000 missing segments. In this
case after the new load is completed on both the main table and SI table,
the SILoadEventListenerForFailedSegments will try to load all the missing
10000 segments back to the SI table. Since there are a lot of segments to be
reloaded in the SI table, this step will block the next load command for
many hours if not days. To solve this problem please find the 2 step
solution.
Step 1. Add a carbon property which will enable/disable the loading for
missing/failed segments. By default it can be kept true, only when the user
sets it as false this functionality will be disabled.
Step 2. Provide a separate SI repair command thus making the whole
functionality independent of load/insert command. We can provide both table
level as well as segment level command to repair the missing segments.
Example table level command: REPAIR INDEX ON TABLE MAIN_TABLE. This will
check for all the SI table in the main table
Example Segment Level command: REPAIR INDEX ON MAIN_TABLE WHERE SEGMENT.ID
IN (0,1,2,3,4). This will only check for the given segments in all the SI
tables.
Please give your input and suggestions for the above solution.
Rgds
Vikram Ahuja
--
Sent from:
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Locked
