[GitHub] carbondata pull request #2439: [CARBONDATA-2606][Complex DataType Enhancemen...

[qiuchenjian-2]

GitHub user Indhumathi27 opened a pull request:

    https://github.com/apache/carbondata/pull/2439

     [CARBONDATA-2606][Complex DataType Enhancements]Fixed Projection Pushdown when Select filter contains Struct column

    **Problem:**
             If Select filter contains Struct Column which is not in Projection list, then only null value is stored for struct column given in filter and select query result is null.
    **Solution:**
             Pushdown Parent column of corresponding struct type if any struct column is present in Filter list.
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
   
     - [x] Testing done
           Test Case Added
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Indhumathi27/carbondata filter

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2439.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2439
   
----
commit 26df47ef07f6592919f0d9ccd5aa68703360de47
Author: Indhumathi27 <indhumathim27@...>
Date:   2018-07-03T11:10:55Z

    Fixed Projection Pushdown when Select filter contains Struct column

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5562/



---

[qiuchenjian-2]

Github user Indhumathi27 commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    Retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6737/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5571/



---

[qiuchenjian-2]

Github user Indhumathi27 commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    Retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6747/



---

[qiuchenjian-2]

Github user Indhumathi27 commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    Retest sdv please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5606/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5610/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5597/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6775/



---

[qiuchenjian-2]

Github user kunal642 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2439#discussion_r200281974
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDatasourceHadoopRelation.scala ---
    @@ -69,84 +69,90 @@ case class CarbonDatasourceHadoopRelation(
       override def schema: StructType = tableSchema.getOrElse(carbonRelation.schema)
     
       def buildScan(requiredColumns: Array[String],
    +      filterComplex: Seq[org.apache.spark.sql.catalyst.expressions.Expression],
           projects: Seq[NamedExpression],
           filters: Array[Filter],
           partitions: Seq[PartitionSpec]): RDD[InternalRow] = {
         val filterExpression: Option[Expression] = filters.flatMap { filter =>
           CarbonFilters.createCarbonFilter(schema, filter)
         }.reduceOption(new AndExpression(_, _))
     
    -    var parentColumn = new ListBuffer[String]
    -    // In case of Struct or StructofStruct Complex type, get the project column for given
    -    // parent/child field and pushdown the corresponding project column. In case of Array,
    -    // ArrayofStruct or StructofArray, pushdown parent column
    -    var reqColumns = projects.map {
    -      case a@Alias(s: GetStructField, name) =>
    -        var arrayTypeExists = false
    -        var ifGetArrayItemExists = s
    -        breakable({
    -          while (ifGetArrayItemExists.containsChild != null) {
    -            if (ifGetArrayItemExists.child.isInstanceOf[AttributeReference]) {
    -              arrayTypeExists = s.childSchema.toString().contains("ArrayType")
    -              break
    -            } else {
    -              if (ifGetArrayItemExists.child.isInstanceOf[GetArrayItem]) {
    -                arrayTypeExists = true
    +    val projection = new CarbonProjection
    +
    +    val complexFilterExists = filterComplex.map(col =>
    +      col.map(_.isInstanceOf[GetStructField]))
    +
    +    if (!complexFilterExists.exists(f => f.contains(true))) {
    --- End diff --
   
    add detailed comment on why this check was needed


---

[qiuchenjian-2]

Github user kunal642 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2439#discussion_r200281753
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala ---
    @@ -636,4 +636,18 @@ class TestComplexDataType extends QueryTest with BeforeAndAfterAll {
         sql("select b.c[0],a[0][0] from test").show(false)
       }
     
    +  test("test struct complex type with filter")
    +  {
    --- End diff --
   
    move to previous line


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6827/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5624/



---

[qiuchenjian-2]

Github user Indhumathi27 commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    Retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5633/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6842/



---

[qiuchenjian-2]

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2439
 
    restest this please


---