Apache CarbonData Dev Mailing List archive › Apache CarbonData JIRA issues

[GitHub] carbondata pull request #1835: [CARBONDATA-2057] Support specifying path whe...

Classic

List

Threaded

15 messages Options

qiuchenjian-2

[GitHub] carbondata pull request #1835: [CARBONDATA-2057] Support specifying path whe...

GitHub user jackylk opened a pull request:

https://github.com/apache/carbondata/pull/1835

[CARBONDATA-2057] Support specifying path when creating pre-aggregate table

When creating datamap of pre-aggreagate table, user should be able to specify the store location of it.
User can use "path" property:
```
CREATE DATAMAP agg ON TABLE main
USING 'preaggregate'
DMPROPERTIES ('path'='datamap_storage_path')
AS SELECT ...
```

- [ ] Any interfaces changed?

- [ ] Any backward compatibility impacted?

- [ ] Document update required?

- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jackylk/incubator-carbondata datamap_location

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1835.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1835

----
commit b76bf7988a1c56b6657668784c53af44862645fa
Author: Jacky Li <jacky.likun@...>
Date: 2018-01-02T15:46:14Z

[CARBONDATA-1968] Add external table support

This PR adds support for creating external table with existing carbondata files, using Hive syntax.
CREATE EXTERNAL TABLE tableName STORED BY 'carbondata' LOCATION 'path'

This closes #1749

commit 65d07cc86020a858bbd611893f6e991601b63f96
Author: Jacky Li <jacky.likun@...>
Date: 2018-01-06T12:28:44Z

[CARBONDATA-1992] Remove partitionId in CarbonTablePath

In CarbonTablePath, there is a deprecated partition id which is always 0, it should be removed to avoid confusion.

This closes #1765

commit 3e1da7c3be6298c620214c7c505b91e6c4596ff8
Author: SangeetaGulia <sangeeta.gulia@...>
Date: 2017-09-21T09:26:26Z

[CARBONDATA-1827] S3 Carbon Implementation

1.Provide support for s3 in carbondata.
2.Added S3Example to create carbon table on s3.
3.Added S3CSVExample to load carbon table using csv from s3.

This closes #1805

commit d1ae835671db60dd3c96f9b87489006857a31837
Author: Jacky Li <jacky.likun@...>
Date: 2018-01-19T06:48:36Z

add datamap path

----

---

qiuchenjian-2

[GitHub] carbondata pull request #1835: [CARBONDATA-2057] Support specifying path whe...

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1835#discussion_r162550458

--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/TestDataMapCommand.scala ---
@@ -207,6 +209,26 @@ class TestDataMapCommand extends QueryTest with BeforeAndAfterAll {
Seq(Row(1, 31), Row(2, 27), Row(3, 70), Row(4, 55)))
}

+ test("create pre-agg table with path") {
+ sql("drop table if exists maintbl_preagg")
+ sql("drop table if exists maintbl ")
+ val path = "./_pre-agg_test"
+ try {
+ sql("create table maintbl(year int,month int,name string,salary int) stored by 'carbondata' tblproperties('sort_columns'='month,year,name')")
+ sql("insert into maintbl select 10,11,'amy',12")
+ sql("insert into maintbl select 10,11,'amy',12")
+ sql("create datamap preagg on table maintbl " +
+ "using 'preaggregate' " +
+ s"dmproperties ('path'='$path') " +
+ "as select name,avg(salary) from maintbl group by name")
+ assertResult(true)(new File(path).exists())
+ checkAnswer(sql("select name,avg(salary) from maintbl group by name"), Row("amy", 12.0))
--- End diff --

Make sure the data comes from aggregate table, better query aggregate table and check once.

---

qiuchenjian-2