[GitHub] [carbondata] QiangCai opened a new pull request #3592: [Refactory] merge all spark-related modules into a spark module

[GitBox]

QiangCai opened a new pull request #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592
 
 
    ### Why is this PR needed?
    there are too many spark-related modules.
    ### What changes were proposed in this PR?
   1.merge  spark-common, spark2, spark-datasource and spark-common-test into spark module
   2.move carbondata/datamap/mv module to carbondata/mv
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#issuecomment-578199795
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1763/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on issue #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#issuecomment-578396799
 
 
   please rebase

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#issuecomment-581264681
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/4/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#issuecomment-587044358
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2026/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#issuecomment-587045635
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/324/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#issuecomment-587230091
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/325/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#discussion_r380419710
 
 

 ##########
 File path: datamap/bloom/pom.xml
 ##########
 @@ -11,7 +11,7 @@
   </parent>
 
   <artifactId>carbondata-bloom</artifactId>
-  <name>Apache CarbonData :: Bloom Index DataMap</name>
+  <name>Apache CarbonData :: Bloom DataMap</name>
 
 Review comment:
   ```suggestion
     <name>Apache CarbonData :: Bloom Index</name>
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#discussion_r380419770
 
 

 ##########
 File path: datamap/lucene/pom.xml
 ##########
 @@ -11,7 +11,7 @@
   </parent>
 
   <artifactId>carbondata-lucene</artifactId>
-  <name>Apache CarbonData :: Lucene Index DataMap</name>
+  <name>Apache CarbonData :: Lucene DataMap</name>
 
 Review comment:
   ```suggestion
     <name>Apache CarbonData :: Lucene Index</name>
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#discussion_r380419770
 
 

 ##########
 File path: datamap/lucene/pom.xml
 ##########
 @@ -11,7 +11,7 @@
   </parent>
 
   <artifactId>carbondata-lucene</artifactId>
-  <name>Apache CarbonData :: Lucene Index DataMap</name>
+  <name>Apache CarbonData :: Lucene DataMap</name>
 
 Review comment:
   ```suggestion
     <name>Apache CarbonData :: Lucene Index</name>
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#discussion_r380419710
 
 

 ##########
 File path: datamap/bloom/pom.xml
 ##########
 @@ -11,7 +11,7 @@
   </parent>
 
   <artifactId>carbondata-bloom</artifactId>
-  <name>Apache CarbonData :: Bloom Index DataMap</name>
+  <name>Apache CarbonData :: Bloom DataMap</name>
 
 Review comment:
   ```suggestion
     <name>Apache CarbonData :: Bloom Index</name>
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#discussion_r380420125
 
 

 ##########
 File path: datamap/bloom/pom.xml
 ##########
 @@ -11,7 +11,7 @@
   </parent>
 
   <artifactId>carbondata-bloom</artifactId>
-  <name>Apache CarbonData :: Bloom Index DataMap</name>
+  <name>Apache CarbonData :: Bloom DataMap</name>
 
 Review comment:
   Do not modify this. If you want to move this module to another folder, please raise another PR

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#discussion_r380420125
 
 

 ##########
 File path: datamap/bloom/pom.xml
 ##########
 @@ -11,7 +11,7 @@
   </parent>
 
   <artifactId>carbondata-bloom</artifactId>
-  <name>Apache CarbonData :: Bloom Index DataMap</name>
+  <name>Apache CarbonData :: Bloom DataMap</name>
 
 Review comment:
   Suggest to call it Bloom Index, and rename its folder name to 'index'

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#discussion_r380420413
 
 

 ##########
 File path: datamap/bloom/pom.xml
 ##########
 @@ -11,7 +11,7 @@
   </parent>
 
   <artifactId>carbondata-bloom</artifactId>
-  <name>Apache CarbonData :: Bloom Index DataMap</name>
+  <name>Apache CarbonData :: Bloom DataMap</name>
 
 Review comment:
   And you can move secondary index into 'index' folder

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#discussion_r380420452
 
 

 ##########
 File path: datamap/lucene/pom.xml
 ##########
 @@ -11,7 +11,7 @@
   </parent>
 
   <artifactId>carbondata-lucene</artifactId>
-  <name>Apache CarbonData :: Lucene Index DataMap</name>
+  <name>Apache CarbonData :: Lucene DataMap</name>
 
 Review comment:
   same as above comment

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#discussion_r380421058
 
 

 ##########
 File path: pom.xml
 ##########
 @@ -531,14 +528,9 @@
                 <sourceDirectory>${basedir}/geo/src/main/scala</sourceDirectory>
                 <sourceDirectory>${basedir}/processing/src/main/java</sourceDirectory>
                 <sourceDirectory>${basedir}/hadoop/src/main/java</sourceDirectory>
-                <sourceDirectory>${basedir}/integration/spark2/src/main/scala</sourceDirectory>
-                <sourceDirectory>${basedir}/integration/spark2/src/main/spark2.3</sourceDirectory>
-                <sourceDirectory>${basedir}/integration/spark2/src/main/java</sourceDirectory>
-                <sourceDirectory>${basedir}/integration/spark-common/src/main/scala</sourceDirectory>
-                <sourceDirectory>${basedir}/integration/spark-common/src/main/java</sourceDirectory>
-                <sourceDirectory>${basedir}/integration/spark-common/src/main/spark2.3</sourceDirectory>
-                <sourceDirectory>${basedir}/integration/spark-common-test/src/main/scala</sourceDirectory>
-                <sourceDirectory>${basedir}/integration/spark-common-test/src/main/java</sourceDirectory>
+                <sourceDirectory>${basedir}/integration/spark/src/main/scala</sourceDirectory>
+                <sourceDirectory>${basedir}/integration/spark/src/main/spark2.3</sourceDirectory>
 
 Review comment:
   Can we move this folder into a new module call 'spark2.3adapter' module? Then IDE can handle the profile more easily

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#issuecomment-587237962
 
 
   Build Failed  with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/326/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#issuecomment-587245850
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2028/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#issuecomment-587259328
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/327/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3592: [Refactory] merge all spark-related modules into a spark module
URL: https://github.com/apache/carbondata/pull/3592#issuecomment-587268966
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2029/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services