[GitHub] carbondata pull request #2520: [CARBONDATA-2750] Added Documentation for Loc...

[qiuchenjian-2]

GitHub user praveenmeenakshi56 opened a pull request:

    https://github.com/apache/carbondata/pull/2520

    [CARBONDATA-2750] Added Documentation for Local Dictionary Support

    ### What has been added?
    Documentation for Local Dictionary Support has been added.
     - [ ] Any interfaces changed?
     NA
     - [ ] Any backward compatibility impacted?
     NA
     - [ ] Document update required?
    Document has been added in this PR.
     - [ ] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
     NA    
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
    NA


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/praveenmeenakshi56/carbondata local_dict_doc

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2520.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2520
   
----
commit d625284b0c8a371b92f41d722fc299e69412f426
Author: praveenmeenakshi56 <praveenmeenakshi56@...>
Date:   2018-07-18T06:07:29Z

    Added Documentation for Local Dictionary Support

----


---

[qiuchenjian-2]

Github user sgururajshetty commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2520#discussion_r203277284
 
    --- Diff: docs/data-management-on-carbondata.md ---
    @@ -122,6 +122,45 @@ This tutorial is going to introduce all commands and data operations on CarbonDa
          TBLPROPERTIES ('streaming'='true')
          ```
     
    +  - **Local Dictionary Configuration**
    +  
    +  Local Dictionary is generated only for no-dictionary string/varchar datatype columns. It helps in:
    +  1. Getting more compression on dimension columns with less cardinality.
    +  2. Filter queries and full scan queries on No-dictionary columns with local dictionary will be faster as filter will be done on encoded data.
    +  3. Reducing the store size and memory footprint as only unique values will be stored as part of local dictionary and corresponding data will be stored as encoded data.
    +
    +       By default, Local Dictionary will be enabled and generated for all no-dictionary string/varchar datatype columns.
    --- End diff --
   
    Convert this into table
   
    | Properties | Default Value | Description |
   
    The **description** should satisfy the following points:
    a. What does this parameter do?
    b. In what scenario the user needs to configure this parameter?
    c. Are there any benefits in configuring this parameter?
    d. What is the default value?
    e. What is the value range if any?
    f. Are there any limitations?
    g. Any key information to be highlighted?


---

[qiuchenjian-2]

Github user sgururajshetty commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2520#discussion_r203281996
 
    --- Diff: docs/data-management-on-carbondata.md ---
    @@ -291,6 +330,11 @@ This tutorial is going to introduce all commands and data operations on CarbonDa
          ALTER TABLE carbon ADD COLUMNS (a1 INT, b1 STRING) TBLPROPERTIES('DEFAULT.VALUE.a1'='10')
          ```
     
    +     Users can specify which columns to include and exclude for local dictionary generation after adding   new columns. These will be appended with the already   existing local dictionary include and exclude          columns of main table respectively.
    --- End diff --
   
    check the spacing between words


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7275/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6044/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5905/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7309/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6074/



---

[qiuchenjian-2]

Github user praveenmeenakshi56 commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5921/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7318/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6082/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7345/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6109/



---

[qiuchenjian-2]

Github user sgururajshetty commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    LGTM


---

[qiuchenjian-2]

Github user praveenmeenakshi56 commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7384/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6145/



---

[qiuchenjian-2]

Github user kumarvishal09 commented on the issue:

    https://github.com/apache/carbondata/pull/2520
 
    LGTM


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2520#discussion_r204969382
 
    --- Diff: docs/data-management-on-carbondata.md ---
    @@ -333,6 +373,20 @@ This tutorial is going to introduce all commands and data operations on CarbonDa
          ```
          ALTER TABLE test_db.carbon CHANGE a1 a1 DECIMAL(18,2)
          ```
    +   - **SET and UNSET for Local Dictionary Properties**
    +  
    +      When set command is used, all the newly set properties will override the corresponding old properties if exists.
    +    
    +      Example to SET Local Dictionary Properties:
    +       ```
    +      ALTER TABLE tablename SET TBLPROPERTIES('LOCAL_DICTIONARY_ENABLE'='false',’LOCAL_DICTIONARY_THRESHOLD'='1000','LOCAL_DICTIONARY_INCLUDE'='column1','LOCAL_DICTIONARY_EXCLUDE'='column2')
    --- End diff --
   
    `’` before `LOCAL_DICTIONARY_THRESHOLD` is wrong


---