[GitHub] carbondata pull request #2578: Fix Dictionary_Include for ComplexDataType

[qiuchenjian-2]

GitHub user Indhumathi27 opened a pull request:

    https://github.com/apache/carbondata/pull/2578

    Fix Dictionary_Include for ComplexDataType

    Why this PR for?
    Check if a column is given as Dictionary_Include and fill cardinality for that column
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
   
     - [ ] Testing done
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Indhumathi27/carbondata dict_fix

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2578.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2578
   
----
commit bab79be62af7ffecb9dc1fcfe520fd0288c0281a
Author: Indhumathi27 <indhumathim27@...>
Date:   2018-07-30T08:48:44Z

    Fix Dictionary_Include for ComplexDataType

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7607/



---

[qiuchenjian-2]

Github user kunal642 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2578#discussion_r206091924
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
    @@ -172,7 +191,7 @@ public void setSurrogateIndex(int surrIndex) {
     
       @Override
       public void fillCardinality(List<Integer> dimCardWithComplex) {
    -    if (children.getIsColumnDictionary()) {
    +    if (this.isDictionaryColumn) {
    --- End diff --
   
    return isDictionaryColumn from getIsColumnDictionary() also.


---

[qiuchenjian-2]

Github user kunal642 commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    Please add a test case to verify the fix



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7615/



---

[qiuchenjian-2]

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    retest this please



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7628/



---

[qiuchenjian-2]

Github user Indhumathi27 commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    Retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7638/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6358/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6378/



---

[qiuchenjian-2]

Github user kunal642 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2578#discussion_r206396378
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/model/LoadOption.java ---
    @@ -236,7 +236,7 @@
           }
         }
     
    -    if (carbonLoadModel.isCarbonTransactionalTable() && !CarbonDataProcessorUtil
    +    if (hadoopConf != null && !CarbonDataProcessorUtil
    --- End diff --
   
    why hadoopConf null check is needed?


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6071/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7658/



---

[qiuchenjian-2]

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    retest this please



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6392/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7672/



---

[qiuchenjian-2]

Github user kunal642 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2578#discussion_r206531160
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestNonTransactionalCarbonTable.scala ---
    @@ -52,10 +52,11 @@ import org.apache.carbondata.core.datastore.page.encoding.DefaultEncodingFactory
     import org.apache.carbondata.core.metadata.ColumnarFormatVersion
     import org.apache.carbondata.core.metadata.datatype.DataTypes
     import org.apache.carbondata.core.util.{CarbonProperties, CarbonUtil, DataFileFooterConverterV3}
    +import org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException
     import org.apache.carbondata.sdk.file._
     
     
    -class TestNonTransactionalCarbonTable extends QueryTest with BeforeAndAfterAll {
    +class TestNonTransactiogitnalCarbonTable extends QueryTest with BeforeAndAfterAll {
    --- End diff --
   
    Please revert this change


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6402/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2578
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7684/



---