[GitHub] [carbondata] Indhumathi27 opened a new pull request #3541: [WIP]Timeseries query is not hitting datamap if granularity in query is given case insensitive

[GitBox]

Indhumathi27 opened a new pull request #3541: [WIP]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541
 
 
   Be sure to do all of the following checklist to help us incorporate
   your contribution quickly and easily:
   
    - [ ] Any interfaces changed?
   
    - [ ] Any backward compatibility impacted?
   
    - [ ] Document update required?
   
    - [ ] Testing done
           Please provide details on
           - Whether new unit test cases have been added or why no new tests are required?
           - How it is tested? Please attach test report.
           - Is it a performance related change? Please attach the performance test report.
           - Any additional information to help reviewers in testing this change.
         
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3541: [WIP]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-569406901
 
 
   Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1318/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3541: [WIP]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-569412192
 
 
   Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1331/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3541: [WIP]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-569417803
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1342/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-569666573
 
 
   Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1359/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-569677750
 
 
   Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1369/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-569680532
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1380/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on issue #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-570565503
 
 
   retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-570594696
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1449/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

Indhumathi27 commented on issue #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-574549797
 
 
   retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-574579295
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1644/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

Indhumathi27 commented on issue #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-577097169
 
 
   retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-577129244
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1736/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-577539536
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1751/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

Indhumathi27 commented on issue #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#issuecomment-579086990
 
 
   @akashrn5 Please review

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#discussion_r373316719
 
 

 ##########
 File path: datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/DefaultMatchMaker.scala
 ##########
 @@ -76,8 +82,16 @@ abstract class DefaultMatchPattern extends MatchPattern[ModularPlan] {
                   qualifier = a.qualifier)
               }.getOrElse(a)
           case a: Expression =>
-            aliasMapExp
+            var attribute = aliasMapExp
               .get(a)
+            if (attribute.isEmpty) {
 
 Review comment:
   better to give an example in comment, that in which case the attribute is empty

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#discussion_r373317237
 
 

 ##########
 File path: datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/Utils.scala
 ##########
 @@ -519,4 +521,73 @@ object Utils extends PredicateHelper {
     }
   }
 
+  /**
+   * Check's if timeseries udf function exists. If exists, compare literal with case insensitive
+   * value
+   */
+  def isExpressionMatchesUDF(exp: Expression, exprList: Seq[Expression]): Boolean = {
+    exp match {
+      case Alias(s: ScalaUDF, _) if s.function.isInstanceOf[TimeSeriesFunction] =>
+        val children = s.children
+        val finalExpr = exprList.filter(a1 =>
+          a1.isInstanceOf[Alias] && a1.asInstanceOf[Alias].child.isInstanceOf[ScalaUDF] &&
+          a1.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].function.
+            isInstanceOf[TimeSeriesFunction])
+        finalExpr.exists(f => {
+          val finalExp = f.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].children
+          finalExp.head.semanticEquals(children.head) &&
+          finalExp.last.asInstanceOf[Literal].toString().equalsIgnoreCase(
+            children.last.asInstanceOf[Literal].toString())
+        })
+      case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+        val children = s.children
+        var expList: Seq[Expression] = Seq.empty
+        exprList foreach {
+          case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+            expList = expList.+:(s)
+          case Alias(s: ScalaUDF, _) if s.function.isInstanceOf[TimeSeriesFunction] =>
+            expList = expList.+:(s.asInstanceOf[Expression])
+          case _ =>
+        }
+        expList.exists(f => {
+          val finalExp = f.asInstanceOf[ScalaUDF].children
+          finalExp.head.semanticEquals(children.head) &&
+          finalExp.last.asInstanceOf[Literal].toString().equalsIgnoreCase(
+            children.last.asInstanceOf[Literal].toString())
+        })
+      case exp: Expression =>
+        val newExp = exp.transform {
+          case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+            getTransformedTimeSeriesUDF(s)
+          case other => other
+        }
+        val newExprList = exprList map { expr =>
+          expr.transform {
+            case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+              getTransformedTimeSeriesUDF(s)
+            case other => other
+          }
+        }
+        newExprList.exists(_.semanticEquals(newExp))
+      case _ => false
+    }
+  }
+
+  def getTransformedTimeSeriesUDF(s: ScalaUDF): Expression = {
+    s.transform {
+      case l: Literal =>
+        Literal(l.toString().toLowerCase, l.dataType)
+    }
+  }
+
+  def isExpressionMatchesUDF(exp: Expression, expr: Expression): Boolean = {
 
 Review comment:
   add comment

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#discussion_r373319253
 
 

 ##########
 File path: datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/Utils.scala
 ##########
 @@ -519,4 +521,73 @@ object Utils extends PredicateHelper {
     }
   }
 
+  /**
+   * Check's if timeseries udf function exists. If exists, compare literal with case insensitive
+   * value
+   */
+  def isExpressionMatchesUDF(exp: Expression, exprList: Seq[Expression]): Boolean = {
+    exp match {
+      case Alias(s: ScalaUDF, _) if s.function.isInstanceOf[TimeSeriesFunction] =>
+        val children = s.children
+        val finalExpr = exprList.filter(a1 =>
+          a1.isInstanceOf[Alias] && a1.asInstanceOf[Alias].child.isInstanceOf[ScalaUDF] &&
+          a1.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].function.
+            isInstanceOf[TimeSeriesFunction])
+        finalExpr.exists(f => {
+          val finalExp = f.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].children
+          finalExp.head.semanticEquals(children.head) &&
+          finalExp.last.asInstanceOf[Literal].toString().equalsIgnoreCase(
+            children.last.asInstanceOf[Literal].toString())
+        })
+      case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+        val children = s.children
+        var expList: Seq[Expression] = Seq.empty
 
 Review comment:
   rename to more meaningful one, here `expList` and `exprList` are both confusing

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#discussion_r373319541
 
 

 ##########
 File path: datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/Utils.scala
 ##########
 @@ -519,4 +521,73 @@ object Utils extends PredicateHelper {
     }
   }
 
+  /**
+   * Check's if timeseries udf function exists. If exists, compare literal with case insensitive
+   * value
+   */
+  def isExpressionMatchesUDF(exp: Expression, exprList: Seq[Expression]): Boolean = {
+    exp match {
+      case Alias(s: ScalaUDF, _) if s.function.isInstanceOf[TimeSeriesFunction] =>
+        val children = s.children
+        val finalExpr = exprList.filter(a1 =>
+          a1.isInstanceOf[Alias] && a1.asInstanceOf[Alias].child.isInstanceOf[ScalaUDF] &&
+          a1.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].function.
+            isInstanceOf[TimeSeriesFunction])
+        finalExpr.exists(f => {
+          val finalExp = f.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].children
+          finalExp.head.semanticEquals(children.head) &&
+          finalExp.last.asInstanceOf[Literal].toString().equalsIgnoreCase(
+            children.last.asInstanceOf[Literal].toString())
+        })
+      case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+        val children = s.children
+        var expList: Seq[Expression] = Seq.empty
+        exprList foreach {
+          case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+            expList = expList.+:(s)
+          case Alias(s: ScalaUDF, _) if s.function.isInstanceOf[TimeSeriesFunction] =>
+            expList = expList.+:(s.asInstanceOf[Expression])
+          case _ =>
+        }
+        expList.exists(f => {
+          val finalExp = f.asInstanceOf[ScalaUDF].children
+          finalExp.head.semanticEquals(children.head) &&
+          finalExp.last.asInstanceOf[Literal].toString().equalsIgnoreCase(
+            children.last.asInstanceOf[Literal].toString())
+        })
+      case exp: Expression =>
+        val newExp = exp.transform {
+          case s: ScalaUDF if s.function.isInstanceOf[TimeSeriesFunction] =>
+            getTransformedTimeSeriesUDF(s)
+          case other => other
+        }
+        val newExprList = exprList map { expr =>
 
 Review comment:
   rename to `tranformedExprListWithLowerCase`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3541: [CARBONDATA-3636]Timeseries query is not hitting datamap if granularity in query is given case insensitive
URL: https://github.com/apache/carbondata/pull/3541#discussion_r373319016
 
 

 ##########
 File path: datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/Utils.scala
 ##########
 @@ -519,4 +521,73 @@ object Utils extends PredicateHelper {
     }
   }
 
+  /**
+   * Check's if timeseries udf function exists. If exists, compare literal with case insensitive
+   * value
+   */
+  def isExpressionMatchesUDF(exp: Expression, exprList: Seq[Expression]): Boolean = {
+    exp match {
+      case Alias(s: ScalaUDF, _) if s.function.isInstanceOf[TimeSeriesFunction] =>
+        val children = s.children
+        val finalExpr = exprList.filter(a1 =>
+          a1.isInstanceOf[Alias] && a1.asInstanceOf[Alias].child.isInstanceOf[ScalaUDF] &&
+          a1.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].function.
+            isInstanceOf[TimeSeriesFunction])
+        finalExpr.exists(f => {
+          val finalExp = f.asInstanceOf[Alias].child.asInstanceOf[ScalaUDF].children
 
 Review comment:
   rename it, i think `childExprsOfTimeSeriesUDF`  sounds better

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services