[GitHub] [carbondata] QiangCai opened a new pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan

[GitBox]

CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-786479544


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3755/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-786551057


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5521/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-786552446


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3756/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-788600828


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5098/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-788605734


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3339/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-789545650


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5108/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-789546202


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3349/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

MarvinLitt commented on a change in pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#discussion_r587998973



##########
File path: integration/spark/src/main/scala/org/apache/carbondata/index/TextMatchUDF.scala
##########
@@ -34,13 +32,3 @@ class TextMatchMaxDocUDF extends ((String, Int) => Boolean) with Serializable {
     v1.length > 0
   }
 }

Review comment:
       the same as the Spatial index, if removed, how to use the fun.

##########
File path: integration/spark/src/main/scala/org/apache/carbondata/geo/InPolygonUDF.scala
##########
@@ -59,23 +58,3 @@ class InPolygonRangeListUDF extends ((String, String) => Boolean) with Serializa
     true // Carbon applies the filter. So, Spark do not have to apply filter.
   }
 }

Review comment:
       this filter is use for Spatial index， if remove how about that?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

QiangCai commented on a change in pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#discussion_r588018676



##########
File path: integration/spark/src/main/scala/org/apache/carbondata/geo/InPolygonUDF.scala
##########
@@ -59,23 +58,3 @@ class InPolygonRangeListUDF extends ((String, String) => Boolean) with Serializa
     true // Carbon applies the filter. So, Spark do not have to apply filter.
   }
 }

Review comment:
       1. it will still keep the udf and expression for Spatial index
   
   2. In the old flow,  carbon converts spark expression to spark filter, and then converts spark filter to carbon expression.
   So, after creating carbon expression, this filter will be removed from memory.
   Now in the new flow, it will convert spark expression to carbon expression directly, and not need this filter.
   
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

MarvinLitt commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-791226234


   LGTM


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

MarvinLitt commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-791242231


   merged


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

QiangCai closed pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]