Re: [DISCUSSION] Distributed Index Cache Server

Posted by manishgupta88 on Feb 12, 2019; 4:45pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/DISCUSSION-Distributed-Index-Cache-Server-tp75008p75098.html

+1

1. Add the impacted areas in design document.
2. If any executor goes down then update the index cache to executor mapping
in driver accordingly.
3. Even though the cache would be divided based on index files, the minimum
unit of cache need to be fixed. Example: 1 segment cache should belong to 1
executor only.
4. One possible suggestion: Instead of reading the splits in Carbon driver,
let each executor in index server write to a file and pass the
List[FilePaths] to the driver and let each Carbon Executors to read from
that path. This is the case when the number of splits exceed the threshold.

Regards
Manish Gupta



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/