[GitHub] [carbondata] marchpure opened a new pull request #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface

[GitBox]

marchpure opened a new pull request #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667
 
 
    ### Why is this PR needed?
    Input type of Spark's FilePartition will change from Seq to Array.
   We shall create filepartition by entering ARRAY type, which is compatible with both Array and Seq.
   
    ### What changes were proposed in this PR?
   create filepartition by entering ARRAY type, instead of entering Seq type.
       
    ### Does this PR introduce any user interface change?
    - No
   
    ### Is any new testcase added?
    - No
   
       
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-598178267
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2441/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-598230507
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2444/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-598245231
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/736/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-598325099
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2446/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-598329978
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/738/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

QiangCai commented on a change in pull request #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#discussion_r392010381
 
 

 ##########
 File path: integration/spark/src/main/spark2.3/org/apache/carbondata/spark/adapter/CarbonToSparkAdapter.scala
 ##########
 @@ -17,12 +17,12 @@
 
 package org.apache.carbondata.spark.adapter
 
-import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.{ArrayBuffer, WrappedArray}
 
 import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
 
 object CarbonToSparkAdapter {
   def createFilePartition(index: Int, files: ArrayBuffer[PartitionedFile]) = {
-    FilePartition(index, files.toArray.toSeq)
 
 Review comment:
   better to modify CsvRDDHelper directly and remove adapter/CarbonToSparkAdapter.scala （both 2.3 and 2.4）

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-598997172
 
 
   Build Failed  with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/757/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-599007584
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2465/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-599028184
 
 
   Build Failed  with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/761/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-599028318
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2469/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-599043991
 
 
   Build Failed  with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/762/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-599051312
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2470/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

Indhumathi27 commented on a change in pull request #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#discussion_r394294843
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/carbondata/spark/load/CsvRDDHelper.scala
 ##########
 @@ -94,7 +93,7 @@ object CsvRDDHelper {
     def closePartition(): Unit = {
       if (currentFiles.nonEmpty) {
         val newPartition =
-          CarbonToSparkAdapter.createFilePartition(
+          FilePartition(
 
 Review comment:
   Should be FilePartition( partitions.size, currentFiles.toArray)?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

Indhumathi27 commented on a change in pull request #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#discussion_r394295019
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/carbondata/spark/load/CsvRDDHelper.scala
 ##########
 @@ -20,7 +20,7 @@ package org.apache.carbondata.spark.load
 import java.text.SimpleDateFormat
 import java.util.{Date, Locale}
 
-import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.{ArrayBuffer, WrappedArray}
 
 Review comment:
   Revert this change

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

ajantha-bhat commented on issue #3667: [HOTFIX] Enhanced compatibility with Spark's FilePartition interface
URL: https://github.com/apache/carbondata/pull/3667#issuecomment-605810388
 
 
   @marchpure : please reply to the comments and get it merged.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services