Login  Register

Proposal to enhance sdk reader for concurrent reading

classic Classic list List threaded Threaded
1 message Options Options
Embed post
Permalink
Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Proposal to enhance sdk reader for concurrent reading

Naman Rastogi
8 posts
Dear community,

The current reading through SDK is slow as in CarbonReader, we are reading
the carbondata files sequentially, even though we have individual
CarbonRecordReader for each file. We can parallelize this by adding an API
in CarbonReader class
*List<CarbonReader> readers = CarbonReader.split(numSplits)*
which returns a list of CarbonReaders, which can be used to read
parallelly, as reading each file is independent of other files.

This enables the SDK user to read the files as it is, or in a multithreaded
environment.

Regards
Naman Rastogi