Apache CarbonData Dev Mailing List archive › Apache CarbonData JIRA issues

[GitHub] carbondata pull request #2903: [CARBONDATA-3084]dataload failure fix when fl...

Classic

List

Threaded

25 messages Options

qiuchenjian-2

[GitHub] carbondata pull request #2903: [CARBONDATA-3084]dataload failure fix when fl...

GitHub user akashrn5 opened a pull request:

https://github.com/apache/carbondata/pull/2903

[CARBONDATA-3084]dataload failure fix when float value exceeds the limit

### Problem:
when the float value exceeds the range and we try to insert that data, data load fails.

### Analysis:
when the value exceeds the range, the max is set as `Infinity`, so the decimal count of that value will be 0, so when decimal count is zero we go for CodecByAlgorithmForIntegral, so it fails

### Solution:
when the value exceeds, and decimal count is zero , source datatype is float, then select DirectCompressCodec

### How this tested:
test cases are added to validate the load and data

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

- [ ] Any interfaces changed?

- [ ] Any backward compatibility impacted?

- [ ] Document update required?

- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/akashrn5/incubator-carbondata float

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2903.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2903

----
commit 798592441f77300dd89b1dec4dd51c15bc6b2c07
Author: akashrn5 <akashnilugal@...>
Date: 2018-11-06T07:06:24Z

dataload failure fix when float value exceeds the limit

----

---

qiuchenjian-2

[GitHub] carbondata pull request #2903: [CARBONDATA-3084]dataload failure fix when fl...

Github user kunal642 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2903#discussion_r231038773

--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/DefaultEncodingFactory.java ---
@@ -325,32 +325,42 @@ static ColumnPageCodec selectCodecByAlgorithmForFloating(SimpleStatsResult stats
//Here we should use the Max abs as max to getDatatype, let's say -1 and -10000000, -1 is max,
//but we can't use -1 to getDatatype, we should use -10000000.
double absMaxValue = Math.max(Math.abs(maxValue), Math.abs(minValue));
- if (decimalCount == 0) {
+ if (decimalCount == 0 && srcDataType == DataTypes.FLOAT) {
--- End diff --

no need to check for decimalCount == 0. If dataType is FLOAT call getColumnPageCodec(..)

---

qiuchenjian-2