Ajith S created CARBONDATA-2824:
-----------------------------------
Summary: Distributed CarbonStore
Key: CARBONDATA-2824
URL:
https://issues.apache.org/jira/browse/CARBONDATA-2824 Project: CarbonData
Issue Type: New Feature
Reporter: Ajith S
Assignee: Ajith S
Currently the CarbonStore is very tightly coupled with FileSystem interface and which runs in process JVM like in spark. We can instead make CarbonStore run as a separate service which can be accessed via network/rpc. So as a Followup of CARBONDATA-2688 (CarbonStore Java API and REST API) we can make carbon store distributed
This has some advantages.
1. Distributed CarbonStore can support parallel scanning i.e multiple tasks can start scanning data parallely, which may have a higher parallelism factor than compute layer
2. Distributed CarbonStore can support index service to multiple apps like (spark/ flink/ presto), such that index will be shared to save resource
3. Distributed CarbonStore resource consumption is isolated from application and easily scalable to support higher workloads
4. As a future improvement, Distributed CarbonStore can implement a query cache since it has independent resources
Distributed CarbonStore will have 2 main deployment parts:
Cluster of remote carbon store service
SDK which acts as a client for communication with store
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Locked
