|
|
Dear community,
The current reading through SDK is slow as in CarbonReader, we are reading
the carbondata files sequentially, even though we have individual
CarbonRecordReader for each file. We can parallelize this by adding an API
in CarbonReader class
*List<CarbonReader> readers = CarbonReader.split(numSplits)*
which returns a list of CarbonReaders, which can be used to read
parallelly, as reading each file is independent of other files.
This enables the SDK user to read the files as it is, or in a multithreaded
environment.
Regards
Naman Rastogi
|
Locked
