[DISCUSSION] Refactory on spark related modules

Posted by Jacky Li on Dec 06, 2017; 3:34am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/DISCUSSION-Refactory-on-spark-related-modules-tp29488.html

Hi community,

Currently we have spark-common and spark2 module, and inside spark2 module there are spark2.1 folder and spark2.2 folder.
This creates problems for IDE for developer, and it spark-common is not required since we have drop spark 1.5 support.
So I suggest we:
1. Merge spark2 and spark-common module, name it spark2 module
2. Put spark2.1 folder to a new module called spark2.1  (It mainly holds CarbonSessionsState)
3. Put spark2.2 folder to a new module called spark2.2  (It mainly holds CarbonSessionsState)

When building carbon, user can specify profile to select modules to include for building
And since CarbonSessionState is created by reflection in spark2 module, so this approach will not have cyclic dependency problem

Regards,
Jacky