[DISCUSSION] Multi-tenant support by refactoring datamaps

Posted by Indhumathi on Feb 13, 2020; 6:58am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/DISCUSSION-Multi-tenant-support-by-refactoring-datamaps-tp91611.html

Hello all,

Currently, when user creates a datamap, system will store the datamap
metadata in a configurable system folder in HDFS or S3. And also, since we
use same naming conventionsas datamap name for datamapschema,
users cannot create datamap with samename which is already present
in storage.

System folder currently holds the following files,
1. DataMapSchema -> a json file containing schema for one datamap.
2. DataMapStatus -> status for each datamap

In cloud scenarios, when one user creates SYSTEM_FOLDER and stores metadata
for materalized views and index datamap's such as bloom and lucene, other
user's are not able to access the SYSTEM_FOLDER.

In order to support multi-tenancy for datamaps, i am planning to move
system_folder under each database level, so that users can access it.
As system folder is moved across database folder,users can create datamap
with same name under different databases.

Datamaps will be saved to database folder specified while creating datamap.

Any suggestions/inputs from the community is appreciated.

Thanks
Indhumathi