[GitHub] incubator-carbondata pull request #339: [CARBONDATA-429][WIP] Remove unneces...

[qiuchenjian-2]

GitHub user jackylk opened a pull request:

    https://github.com/apache/incubator-carbondata/pull/339

    [CARBONDATA-429][WIP] Remove unnecessary file name check in dictionary cache

    In dictionary cache, there are currently necessary file name check for each column, which increase the number of HDFS interactions unnecessarily.
    In this PR, file name check is replaced by primitive type check, and returns null if it is a complex type.
    Dictionary cache support for complex type is WIP.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jackylk/incubator-carbondata dictionary

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/339.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #339
   
----

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/339
 
    Build finished.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user piaoyats commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/339#discussion_r92331735
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java ---
    @@ -167,12 +167,9 @@ private Dictionary getDictionary(
           DictionaryColumnUniqueIdentifier dictionaryColumnUniqueIdentifier)
           throws CarbonUtilException {
         Dictionary reverseDictionary = null;
    -    // create column dictionary info object only if dictionary and its
    -    // metadata file exists for a given column identifier
    -    if (!isFileExistsForGivenColumn(dictionaryColumnUniqueIdentifier)) {
    -      throw new CarbonUtilException(
    -          "Either dictionary or its metadata does not exist for column identifier :: "
    -              + dictionaryColumnUniqueIdentifier.getColumnIdentifier());
    +    // create column dictionary info object only if it is primitive type.
    +    if (dictionaryColumnUniqueIdentifier.getDataType().isComplexType()) {
    +      return null;
    --- End diff --
   
    I am not sure this condition check is right or not? Maybe we still need to do dictionary encoding for complex type?
    @QiangCai  pls review this logic


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/339#discussion_r92332659
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java ---
    @@ -167,12 +167,9 @@ private Dictionary getDictionary(
           DictionaryColumnUniqueIdentifier dictionaryColumnUniqueIdentifier)
           throws CarbonUtilException {
         Dictionary reverseDictionary = null;
    -    // create column dictionary info object only if dictionary and its
    -    // metadata file exists for a given column identifier
    -    if (!isFileExistsForGivenColumn(dictionaryColumnUniqueIdentifier)) {
    -      throw new CarbonUtilException(
    -          "Either dictionary or its metadata does not exist for column identifier :: "
    -              + dictionaryColumnUniqueIdentifier.getColumnIdentifier());
    +    // create column dictionary info object only if it is primitive type.
    +    if (dictionaryColumnUniqueIdentifier.getDataType().isComplexType()) {
    +      return null;
    --- End diff --
   
    We will not invoke getDictionary() method on  complex type column directly.  For complex type column, we call getDictionary() method on primitive type sub-column.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user piaoyats commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/339#discussion_r92345835
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java ---
    @@ -167,12 +167,9 @@ private Dictionary getDictionary(
           DictionaryColumnUniqueIdentifier dictionaryColumnUniqueIdentifier)
           throws CarbonUtilException {
         Dictionary reverseDictionary = null;
    -    // create column dictionary info object only if dictionary and its
    -    // metadata file exists for a given column identifier
    -    if (!isFileExistsForGivenColumn(dictionaryColumnUniqueIdentifier)) {
    -      throw new CarbonUtilException(
    -          "Either dictionary or its metadata does not exist for column identifier :: "
    -              + dictionaryColumnUniqueIdentifier.getColumnIdentifier());
    +    // create column dictionary info object only if it is primitive type.
    +    if (dictionaryColumnUniqueIdentifier.getDataType().isComplexType()) {
    +      return null;
    --- End diff --
   
    So, here logic is right.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user piaoyats commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/339
 
    LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user chenliang613 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/339
 
    test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/339
 
    Build Failed  with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/191/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/339
 
    Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/230/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user kumarvishal09 commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/339#discussion_r93005864
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java ---
    @@ -167,12 +167,9 @@ private Dictionary getDictionary(
           DictionaryColumnUniqueIdentifier dictionaryColumnUniqueIdentifier)
           throws CarbonUtilException {
         Dictionary reverseDictionary = null;
    -    // create column dictionary info object only if dictionary and its
    -    // metadata file exists for a given column identifier
    -    if (!isFileExistsForGivenColumn(dictionaryColumnUniqueIdentifier)) {
    -      throw new CarbonUtilException(
    -          "Either dictionary or its metadata does not exist for column identifier :: "
    -              + dictionaryColumnUniqueIdentifier.getColumnIdentifier());
    +    // create column dictionary info object only if it is primitive type.
    +    if (dictionaryColumnUniqueIdentifier.getDataType().isComplexType()) {
    --- End diff --
   
    Do we need this check?? for complex data type column like array and struct no need to invoke this class, as dictionary will be generated only for primitive type column.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/339#discussion_r93011891
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java ---
    @@ -167,12 +167,9 @@ private Dictionary getDictionary(
           DictionaryColumnUniqueIdentifier dictionaryColumnUniqueIdentifier)
           throws CarbonUtilException {
         Dictionary reverseDictionary = null;
    -    // create column dictionary info object only if dictionary and its
    -    // metadata file exists for a given column identifier
    -    if (!isFileExistsForGivenColumn(dictionaryColumnUniqueIdentifier)) {
    -      throw new CarbonUtilException(
    -          "Either dictionary or its metadata does not exist for column identifier :: "
    -              + dictionaryColumnUniqueIdentifier.getColumnIdentifier());
    +    // create column dictionary info object only if it is primitive type.
    +    if (dictionaryColumnUniqueIdentifier.getDataType().isComplexType()) {
    --- End diff --
   
    ok, I will change it to use assert


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/339
 
    Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/236/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-carbondata/pull/339


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---