Apache CarbonData Dev Mailing List archive › Apache CarbonData JIRA issues

[GitHub] carbondata pull request #2603: [Documentation] Editorial review comment fixe...

Classic

List

Threaded

21 messages Options

qiuchenjian-2

[GitHub] carbondata pull request #2603: [Documentation] Editorial review comment fixe...

GitHub user sgururajshetty opened a pull request:

https://github.com/apache/carbondata/pull/2603

[Documentation] Editorial review comment fixed

Minor issues fixed (spelling, syntax, and missing info)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sgururajshetty/carbondata editorial_review1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2603.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2603

----
commit 529f80dda6db3ce34e0baf766b03a9a13190b286
Author: sgururajshetty <sgururajshetty@...>
Date: 2018-07-25T12:44:07Z

Documentation for support for COLUMN_META_CACHE in create table and alter table properties

commit d816aaa7a89155b3579906f960ed6a0ba4d4a59f
Author: sgururajshetty <sgururajshetty@...>
Date: 2018-07-25T12:48:43Z

Documentation to support for CACHE_LEVEL in create table and alter table properties

commit 8ac243f8e9cff8359b6064352deb823eda7b9835
Author: sgururajshetty <sgururajshetty@...>
Date: 2018-07-25T13:24:52Z

Review comment fixed

commit 98501d35cfd110bcb9e75eb02628f3bce0c0f4ab
Author: sgururajshetty <sgururajshetty@...>
Date: 2018-07-25T13:26:58Z

review comment fixed

commit 62caf822cbcde1e519501c1d5db3c5cfc05fbd63
Author: Indhumathi27 <indhumathim27@...>
Date: 2018-07-21T10:46:21Z

[CARBONDATA-2606]Fix Complex array Pushdown and block auto merge compaction

1.Check for if Complex Column contains ArrayType at n levels and add parent to projection if contains array.
2.Block Auto merge compaction for table containing complex datatype columns.
3.Fix Decimal Datatype scale and precision with two level struct type
4.Fix Dictionary Include for ComplexDataType
- If other complex columns other than first complex column is given in dictionary include, then its insertion fails.
5.Fix BadRecord and dateformat for Complex primitive type-DATE

This closes #2535

commit d287a102b5c96e54261ac00c77038a1a56161fe9
Author: kumarvishal09 <kumarvishal1802@...>
Date: 2018-07-24T14:40:54Z

[CARBONDATA-2779]Fixed filter query issue in case of V1/v2 format store

Problem:
Filter query is failing for V1/V2 carbondata store

Root Cause:
in V1 store measure min max was not added in blockminmaxindex in executor when filter is applied min max pruning is failing with array index out of cound exception

Solution:
Need to add min max for measure column same as already handled in driver block pruning

This closes #2550

commit b08745f68624ff066e0b23a41ce12d4a99618ac5
Author: Manhua <kevinjmh@...>
Date: 2018-07-25T08:51:49Z

[CARBONDATA-2783][BloomDataMap][Doc] Update document for bloom filter datamap

add example for enable/disable datamap

This closes #2554

commit 964d26866468df6be130e9d65d339439cb4cf3b0
Author: praveenmeenakshi56 <praveenmeenakshi56@...>
Date: 2018-07-25T15:31:37Z

[CARBONDATA-2750] Added Documentation for Local Dictionary Support

Added Documentation for Local Dictionary Support

This closes #2520

commit 1fa9f64d70123d0bc988427a34c0750283f5daae
Author: BJangir <babulaljangir111@...>
Date: 2018-07-23T16:44:12Z

[CARBONDATA-2772] Size based dictionary fallback is failing even threshold is not reached.

Issue:- Size Based Fallback happened even threshold is not reached.
RootCause:- Current size calculation is wrong. it is calculated for each data. instead of generated dictionary data .

Solution :- Current size should be calculated only for generated dictionary data.

This closes #2542

commit eae5817e56a20aecb7694c8d387dbb05b96e1045
Author: kunal642 <kunalkapoor642@...>
Date: 2018-07-24T10:42:54Z

[CARBONDATA-2778]Fixed bug when select after delete and cleanup is showing empty records

Problem: In case if delete operation when it is found that the data being deleted is leading to a state where one complete block data is getting
deleted. In that case the status if that block is marked for delete and during the next delete operation run the block is deleted along with its
carbonIndex file. The problem arises due to deletion of carbonIndex file because for multiple blocks there can be one carbonIndex file as one
carbonIndex file represents one task.

Solution: Do not delete the carbondata and carbonIndex file. After compaction it will automatically take care of deleting the stale data and stale segments.

This closes #2548

commit 6d6874a11482a8aa79f2280f6572e84b5e3cbc93
Author: dhatchayani <dhatcha.official@...>
Date: 2018-07-25T09:11:58Z

[CARBONDATA-2753][Compatibility] Row count of page is calculated wrong for old store(V2 store)

Row count of page is calculated wrong for V2 store.

commit b6f5af6af96140876ec10ff09c3313d9b35ceb36
Author: Sssan520 <liangap2008@...>
Date: 2018-07-25T11:36:00Z

[CARBONDATA-2782]delete dead code in class 'CarbonCleanFilesCommand'

The variables(dmsãindexDms) in function processMetadata are nerver used.

This closes #2557

commit cf6bb46d89a9630209ef7f7e797ca8e156369206
Author: ajantha-bhat <ajanthabhat@...>
Date: 2018-07-24T07:03:47Z

[CARBONDATA-2775] Adaptive encoding fails for Unsafe OnHeap. if, target datatype is SHORT_INT

problem:
[CARBONDATA-2775] Adaptive encoding fails for Unsafe OnHeap if, target data type is SHORT_INT

solution: If ENABLE_OFFHEAP_SORT = false, in carbon property. UnsafeFixLengthColumnPage.java will use different compress logic. Not the raw compression. In that case, for SHORT_INT data type , conversion need to handle.

This closes #2546

commit 1fd3889cfec902e30cdcb9af6ed7b7f0df0f20e4
Author: mohammadshahidkhan <mohdshahidkhan1987@...>
Date: 2018-07-25T06:44:38Z

[HOTFIX] Fixed random test failure

Fixed random test failure

This closes #2553

commit 0f1aabb87b81670554188b698fc9fb4c598d8942
Author: Jatin <jatin.demla@...>
Date: 2018-07-25T19:12:50Z

[HOTFIX] CreateDataMapPost Event was skipped in case of preaggregate datamap

CreateDataMapPost Event was skipped in case of preaggregate datamap

This closes #2562

commit 9b05edbace9ccca5c7def6d5e4cc157ed0eb0d66
Author: ajantha-bhat <ajanthabhat@...>
Date: 2018-07-25T19:05:36Z

[CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed:Forever blocking wait with more than 21 batch of data

problem: [CARBONDATA-2784]
[SDK writer] Forever blocking wait with more than 21 batch of data, when consumer is dead due to data loading exception (bad record / out of memory)

root cause:
When the consumer is dead due to data loading exception, writer will be forcefully closed. but queue.clear() cleared only snapshot of entries (10 batches) and close is set to true after that. In between clear() and close = true, If more than 10 batches of data is again put into queue. For 11th batch, queue.put() goes for forever block as consumer is dead.

Solution:
set close = true, before clearing the queue. This will avoid adding more batches to queue from write().

problem [CARBONDATA-2786] NPE when SDK writer tries to write a file

solution and cause:
#2387 , in CarbonProperties.java
After systemLocation = getStorePath(); Null validation missing for systemLocation.
because this can be null in SDK case. As Store location is not applicable for SDK.
All a null validation.

This closes #2561

commit 1c620f83e3f5b1adb28b73f38c40021b2f06e1d0
Author: Indhumathi27 <indhumathim27@...>
Date: 2018-07-27T06:52:25Z

[CARBONDATA-2791]Fix Encoding for Double if exceeds LONG.Max_value

If Factor(decimalcount) * absMaxValue exceeds LONG.MAX_VALUE, then go for direct compression.

This closes #2569

commit 39773ccf01345111bc950d8411646b886111f897
Author: manishgupta88 <tomanishgupta18@...>
Date: 2018-07-27T08:41:11Z

[HOTFIX] Removed file existence check to improve dataMap loading performance

Problem
DataMap loading performance degraded after adding file existence check.

Analysis
When carbonIndex file is read and carbondata file path to its metadata Info map is prepared, file physical existence is getting checked every time which in case of HDFS file system is a namenode call. This degrades the dataMap loading performance. This is done to avoid failures for

Handle IUD scenario where after delete operation carbondata file is deleted but the entry still exists in index file.
Fix
Modified code to check for physical file existence only in case when any IUD operation has happened on the table

This closes #2560

commit a55722c1f6dad54d58e7cd69b9f48bfb3693c2a0
Author: Indhumathi27 <indhumathim27@...>
Date: 2018-07-27T18:09:50Z

[CARBONDATA-2794]Distinct count fails on ArrayOfStruct

This PR fixes Code Generator Error thrown when Select filter contains more than one count of distinct of ArrayofStruct with group by Clause

This closes #2573

commit 6c4a2056f25ddc61d0fa0b205cb65a95c4298549
Author: mohammadshahidkhan <mohdshahidkhan1987@...>
Date: 2018-07-17T11:29:35Z

[CARBONDATA-2749][dataload] In HDFS Empty tablestatus file is written
during datalaod, iud or compaction when disk is full.

Problem:
When a failure happens due to disk full during load, IUD or Compaction,
then while updating the tablestatus file, the tablestaus.tmp file during
atomic file operation remains empty, and in the finally block the empty
tablestaus.tmp file is getting renamed to the actual file.
This leads to empty tablestatus file. Once such problem happens the
tablestatus file can not be retrieved and the already loaded data can be used.

Solution:
If the failures happens during write then the the schema rename in the finally
block must be avoided.

This closes #2517

commit 0a5431c1b645af8820486a55dd33530583ea9ae1
Author: Jimmy Casey <casey.jimmy@...>
Date: 2018-07-29T21:35:27Z

Fixed Spelling

Fixed Spelling

This closes #2584

commit 6fb852ad86cc7ee7dd97af211a6382ec8f4eb750
Author: chenliang613 <chenliang613@...>
Date: 2018-07-27T02:51:45Z

[CARBONDATA-2789] Support Hadoop 2.8.3 eco-system integration

Add hadoop 2.8.3 profile and passed the compile

This closes #2566

commit 937f68cf91bf6708de5c51fe787ea7672b3ee126
Author: ravipesala <ravi.pesala@...>
Date: 2018-07-30T12:45:32Z

[CARBONDATA-2801]Added documentation for flat folder

[CARBONDATA-2801]Added documentation for flat folder

This closes #2582

commit e1abff71c7feba90be0ab338ec3b5aabf9cf9380
Author: ajantha-bhat <ajanthabhat@...>
Date: 2018-07-25T13:51:02Z

[CARBONDATA-2606][Complex DataType Enhancements]Fix Null result if projection column have null primitive column and struct

Problem:
In case if the actual value of the primitive data type is null, by PR#2489, we are moving all the null values to the end of the collected row without considering the data type.

Solution:
Place null in the end of output iff the null value is of complex primitive column.

This closes #2559

commit 917b230b57cff9c350524c81d939b06af6a35cde
Author: akashrn5 <akashnilugal@...>
Date: 2018-07-27T07:01:48Z

[CARBONDATA-2585]disable local dictionary by default

make local dictionary false by default

This closes #2570

commit 5d2d6cf053accdd76bf26943db0a0f09dfc492e1
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-07-26T15:22:58Z

[HotFix][CARBONDATA-2788][BloomDataMap] Fix bugs in incorrect query result with bloom datamap

This PR solve two problems which will affect the correctness of the query on bloom.

Revert PR2539
After review the code, we found that modification in PR2539 is not needed, so we revert that PR.

Bugs in overflow for blocklet count
Carbondata stores blocklet count for each block in byte data type, when a block contains more than 128 blocklets, it will overflow the byte
limits. Here we change the data type to short.

For cache_leve=block, after pruning by main BlockDataMap, the blockletNo in Blocklet is -1, which indicate that the following percedure will scan
the whole block -- all the blocklets in the block. So, when doing intersection with the pruned result from BloomDataMap, we
need to take care of these blocklets. In this implementation, we added the result from BloomDataMap based on the blocklet's existence in BlockDataMap.

This closes #2565

commit ec73f91817062495b085a231279f5133e44b1af8
Author: Indhumathi27 <indhumathim27@...>
Date: 2018-07-30T08:48:44Z

[CARBONDATA-2798] Fix Dictionary_Include for ComplexDataType

Problem1:
Select Filter is throwing BufferUnderFlow Exception as cardinality is filled for Non-Dictionary columns.
Solution:
Check if a complex column has Encoding => Dictionary and fill cardinality for that column only.

Problem2:
Transactional Table is throwing NullPointerException if csv fileheader is not proper.
Solution:
Throw CarbonDataLoadingException if csv fileheader is not proper.

This closes #2578

commit 6fc498055801ab6cf5181b24ee4db47c0510f172
Author: akashrn5 <akashnilugal@...>
Date: 2018-07-30T13:52:29Z

[CARBONDATA-2805] Fix the ordering mismatch of segment numbers during cutom compaction

Problem:
when we have segments from 0 to 6 and i give 1, 2, 3 for custom compaction, then it should create 1.1 as compacted segment, but sometimes
it will create 3.1 as compacted segment which is wrong. This is beacuse custom Segment IDs were passing in hashset and finally inserted in
hashmap, while identifying segments to be merged. hashmap and hashset does not guarantee the insertion order which may lead to missmatch of segment numbers.

Solution:
Use LinkedHashSet and LinkedHashMap which always sure about the insertion order.

This closes #2585

commit 2f828ce2f438de67dabf565be25b9bbad76b49fc
Author: rahul <rahul.kumar@...>
Date: 2018-07-04T14:01:51Z

[CARBONDATA-2625] While BlockletDataMap loading, avoid multiple times listing of files

CarbonReader is very slow for many files as blockletDataMap lists files
of folder while loading each segment. This optimization lists once across segment loads.

This closes #2441

commit 436c8642185c7d262582c55c6c0a077922125be5
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-07-27T03:54:21Z

[CARBONDATA-2790][BloomDataMap]Optimize default parameter for bloomfilter datamap

To provide better query performance for bloomfilter datamap by default,
we optimize bloom_size from 32000 to 640000 and optimize bloom_fpp from
0.01 to 0.00001.

This closes #2567

----

---

qiuchenjian-2