Apache CarbonData Dev Mailing List archive › Apache CarbonData JIRA issues

[GitHub] carbondata pull request #2043: [WIP] PR-2030 sdv fix

Classic

List

Threaded

36 messages Options

qiuchenjian-2

[GitHub] carbondata pull request #2043: [WIP] PR-2030 sdv fix

GitHub user ravipesala opened a pull request:

https://github.com/apache/carbondata/pull/2043

[WIP] PR-2030 sdv fix

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

- [ ] Any interfaces changed?

- [ ] Any backward compatibility impacted?

- [ ] Document update required?

- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ravipesala/incubator-carbondata pr-2030-new

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2043.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2043

----
commit 9086a1b9f2cd6cf1d4d42290a4e3678b01472714
Author: SangeetaGulia <sangeeta.gulia@...>
Date: 2017-09-21T09:26:26Z

[CARBONDATA-1827] S3 Carbon Implementation

1.Provide support for s3 in carbondata.
2.Added S3Example to create carbon table on s3.
3.Added S3CSVExample to load carbon table using csv from s3.

This closes #1805

commit 0c75ab7359ad89a16f749e84bd42416523d5255a
Author: Jacky Li <jacky.likun@...>
Date: 2018-01-02T15:46:14Z

[CARBONDATA-1968] Add external table support

This PR adds support for creating external table with existing carbondata files, using Hive syntax.
CREATE EXTERNAL TABLE tableName STORED BY 'carbondata' LOCATION 'path'

This closes #1749

commit 5663e916fe906675ce8efa320de1ed550315dc00
Author: Jacky Li <jacky.likun@...>
Date: 2018-01-06T12:28:44Z

[CARBONDATA-1992] Remove partitionId in CarbonTablePath

In CarbonTablePath, there is a deprecated partition id which is always 0, it should be removed to avoid confusion.

This closes #1765

commit bd40a0d73d2a7086caaa6773a2c6a1a45e24334c
Author: Jacky Li <jacky.likun@...>
Date: 2018-01-31T16:25:31Z

[REBASE] Solve conflict after rebasing master

commit 92c9f224094581378a681fd1f7b0cb02b923687c
Author: Jacky Li <jacky.likun@...>
Date: 2018-01-30T13:24:04Z

[CARBONDATA-2099] Refactor query scan process to improve readability

Unified concepts in scan process flow:

1.QueryModel contains all parameter for scan, it is created by API in CarbonTable. (In future, CarbonTable will be the entry point for various table operations)
2.Use term ColumnChunk to represent one column in one blocklet, and use ChunkIndex in reader to read specified column chunk
3.Use term ColumnPage to represent one page in one ColumnChunk
4.QueryColumn => ProjectionColumn, indicating it is for projection

This closes #1874

commit f06824e9744a831776b1203c94d4001eef870b14
Author: Jacky Li <jacky.likun@...>
Date: 2018-01-31T08:14:27Z

[CARBONDATA-2025] Unify all path construction through CarbonTablePath static method

Refactory CarbonTablePath:

1.Remove CarbonStorePath and use CarbonTablePath only.
2.Make CarbonTablePath an utility without object creation, it can avoid creating object before using it, thus code is cleaner and GC is less.

This closes #1768

commit f9d15a215adc91077f1a6ca6a456e5fce4bc05eb
Author: sounakr <sounakr@...>
Date: 2017-09-28T10:51:05Z

[CARBONDATA-1480]Min Max Index Example for DataMap

Datamap Example. Implementation of Min Max Index through Datamap. And Using the Index while prunning.

This closes #1359

commit bb5bb00af982831ea83c73a2c437aa4aea8a5422
Author: ravipesala <ravi.pesala@...>
Date: 2017-11-15T14:18:40Z

[CARBONDATA-1544][Datamap] Datamap FineGrain implementation

Implemented interfaces for FG datamap and integrated to filterscanner to use the pruned bitset from FG datamap.
FG Query flow as follows.
1.The user can add FG datamap to any table and implement there interfaces.
2. Any filter query which hits the table with datamap will call prune method of FGdatamap.
3. The prune method of FGDatamap return list FineGrainBlocklet , these blocklets contain the information of block, blocklet, page and rowids information as well.
4. The pruned blocklets are internally wriitten to file and returns only the block , blocklet and filepath information as part of Splits.
5. Based on the splits scanrdd schedule the tasks.
6. In filterscanner we check the datamapwriterpath from split and reNoteads the bitset if exists. And pass this bitset as input to it.

This closes #1471

commit dfbdf3db00cbb488e49d3125b4ec93ff9e0dc9b2
Author: Jatin <jatin.demla@...>
Date: 2018-01-25T11:23:00Z

[CARBONDATA-2080] [S3-Implementation] Propagated hadoopConf from driver to executor for s3 implementation in cluster mode.

Problem : hadoopconf was not getting propagated from driver to the executor that's why load was failing to the distributed environment.
Solution: Setting the Hadoop conf in base class CarbonRDD
How to verify this PR :
Execute the load in the cluster mode It should be a success using location s3.

This closes #1860

commit 586ab70228a09aca0240434327a8d4a4ff14446d
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-08T06:35:14Z

[CARBONDATA-2018][DataLoad] Optimization in reading/writing for sort temp row

Pick up the no-sort fields in the row and pack them as bytes array and skip parsing them during merge sort to reduce CPU consumption

This closes #1792

commit 3fdd5d0f567e8d07cc502202ced7d490fa85e2ad
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-08T06:42:39Z

[CARBONDATA-2023][DataLoad] Add size base block allocation in data loading

Carbondata assign blocks to nodes at the beginning of data loading.
Previous block allocation strategy is block number based and it will
suffer skewed data problem if the size of input files differs a lot.

We introduced a size based block allocation strategy to optimize data
loading performance in skewed data scenario.

This closes #1808

commit d88d5bb940f0fea6e5c8560fc5c8ea3724b95a28
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-08T07:39:45Z

[HotFix][CheckStyle] Fix import related checkstyle

This closes #1952

commit 0bb4aed60a7b60eed49e0b5e618af269c0c03a73
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-08T17:39:20Z

[REBASE] Solve conflict after rebasing master

commit 6216294c1e28c1db05e572f0aac3a991d345e085
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-27T00:51:25Z

[REBASE] resolve conflict after rebasing to master

commit 28b5720fcf1cbd0d4bdf3f04e7b0edd8f9492a8d
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-08T06:35:14Z

[CARBONDATA-2018][DataLoad] Optimization in reading/writing for sort temp row

Pick up the no-sort fields in the row and pack them as bytes array and skip parsing them during merge sort to reduce CPU consumption

This closes #1792

commit 8fe8ab4c078de0ccd218f8ba41352896aebd5202
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-08T06:42:39Z

[CARBONDATA-2023][DataLoad] Add size base block allocation in data loading

Carbondata assign blocks to nodes at the beginning of data loading.
Previous block allocation strategy is block number based and it will
suffer skewed data problem if the size of input files differs a lot.

We introduced a size based block allocation strategy to optimize data
loading performance in skewed data scenario.

This closes #1808

commit 1d85e916f6a0f070960555fb18ee4cd8acbfa315
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-10T02:34:59Z

Revert "[CARBONDATA-2023][DataLoad] Add size base block allocation in data loading"

This reverts commit 6dd8b038fc898dbf48ad30adfc870c19eb38e3d0.

commit 5fccdabfc1cc4656d75e51867dcfcb250c505c91
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-10T11:44:23Z

[CARBONDATA-1997] Add CarbonWriter SDK API

Added a new module called store-sdk, and added a CarbonWriter API, it can be used to write Carbondata files to a specified folder, without Spark and Hadoop dependency. User can use this API in any environment.

This closes #1967

commit 46031a320506ceed10b2134710be6c630c6ee533
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-10T12:11:25Z

Revert "[CARBONDATA-2018][DataLoad] Optimization in reading/writing for sort temp row"

This reverts commit de92ea9a123b17d903f2d1d4662299315c792954.

commit fc31be790934f8a6f59910ab0db453b139cfc7e3
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-11T02:12:10Z

[CARBONDATA-2156] Add interface annotation

InterfaceAudience and InterfaceStability annotation should be added for user and developer

1.InetfaceAudience can be User and Developer
2.InterfaceStability can be Stable, Evolving, Unstable

This closes #1968

commit dcfe73b8b07267369f8c58130f27b75efccb4ee1
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-11T13:37:04Z

[CARBONDATA-2159] Remove carbon-spark dependency in store-sdk module

To make assembling JAR of store-sdk module, it should not depend on carbon-spark module

This closes #1970

commit 503e0d96864173ccfb29e49686f0af3f7edd779f
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-13T01:12:09Z

Support generating assembling JAR for store-sdk module

Support generating assembling JAR for store-sdk module and remove junit dependency

This closes #1976

commit faad967d8d83eabd3e758b081370235e42a3ecee
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-13T02:58:06Z

[CARBONDATA-2091][DataLoad] Support specifying sort column bounds in data loading

Enhance data loading performance by specifying sort column bounds
1. Add row range number during convert-process-step
2. Dispatch rows to each sorter by range number
3. Sort/Write process step can be done concurrently in each range
4. Since all sorttemp files will be written in one folders, we add range
number to the file name to distingush them

Tests added and docs updated

This closes #1953

commit 623a1f93bf50bbbf665d98d71fe2190a77774742
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-20T03:16:53Z

[CARBONDATA-2186] Add InterfaceAudience.Internal to annotate internal interface

This closes #1986

commit ce88eb6a2d6d54acf15a2bdf2a9165ecc9570647
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-24T13:18:17Z

[CARBONDATA-1114][Tests] Fix bugs in tests in windows env

Fix bugs in tests that will cause failure under windows env

This closes #1994

commit 8104735fd66952a531153eb0d3b4db5c9ecc133d
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-27T03:26:30Z

[REBASE] Solve conflict after merging master

commit c7a9f15e2daa0207862aa28c44c51cc7cc081bac
Author: Ravindra Pesala <ravi.pesala@...>
Date: 2017-11-21T10:19:11Z

[CARBONDATA-1543] Supported DataMap chooser and expression for supporting multiple datamaps in single query

This PR supports 3 features.

1.Load datamaps from the DataMapSchema which are created through DDL.
2.DataMap Chooser: It chooses the datamap out of available datamaps based on simple logic. Like if there is filter condition on column1 then for supposing 2 datamaps(1. column1 2. column1+column2) are supporting this column then we choose the datamap which has fewer columns that is the first datamap.
3.Expression support: Based on the filter expressions we convert them to the possible DataMap expressions and do apply expression on it.
For example, there are 2 datamaps available on table1
Datamap1 : column1
Datamap2 : column2
Query: select * from table1 where column1 ='a' and column2 =b
For the above query, we create datamap expression as AndDataMapExpression(Datamap1, DataMap2). So for the above query both the datamaps are included and the output of them will be applied AND condition to improve the performance

This closes #1510

commit f8ded96e659cb1e99cc6ac511d1db7cbc25dddb7
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-22T12:59:59Z

[CARBONDATA-2189] Add DataMapProvider developer interface

Add developer interface for 2 types of DataMap:

1.IndexDataMap: DataMap that leveraging index to accelerate filter query
2.MVDataMap: DataMap that leveraging Materialized View to accelerate olap style query, like SPJG query (select, predicate, join, groupby)
This PR adds support for following logic when creating and dropping the DataMap

This closes #1987

commit 1134431dcfa22fdfc6ff0e9d26688009cf940326
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-26T02:04:51Z

[HOTFIX] Add dava doc for datamap interface

1. Rename some of the datamap interface
2. Add more java doc for all public class of datamap interface

This closes #1998

commit 96ee82b336b178c7a0d5a6d8aa8fd14afc7bc27e
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-27T05:06:02Z

[HOTFIX] Fix timestamp issue in TestSortColumnsWithUnsafe

Fix timestamp issue in TestSortColumnsWithUnsafe

This closes #2001

----

---

qiuchenjian-2