GitHub user xubo245 reopened a pull request:
https://github.com/apache/carbondata/pull/2073 [CARBONDATA-2260] CarbonThriftServer should support store carbon table on S3
CarbonThriftServer should support store carbon table on S3
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [x] Any interfaces changed?
- [x] Any backward compatibility impacted?
- [x] Document update required?
- [x] Testing done
Step 1:
start carbonThriftServer by org.apache.carbondata.examples.CarbonThriftServerExample
User should use themselves storePath, AK, SK and endPoint
Step 2:
start beeline like: beeline -u jdbc:hive2://IP:10000
IP and port should provide
Now it support running with local, HDFS and S3 mode
Test example:
```
root@ecs-909c:~# beeline -u jdbc:hive2://192.168.0.206:10000
Connecting to jdbc:hive2://192.168.0.206:10000
2018-03-17 17:44:29 INFO Utils:310 - Supplied authorities: 192.168.0.206:10000
2018-03-17 17:44:29 INFO Utils:397 - Resolved authority: 192.168.0.206:10000
2018-03-17 17:44:29 INFO HiveConnection:203 - Will try to open client transport with JDBC Uri: jdbc:hive2://192.168.0.206:10000
Connected to: Spark SQL (version 2.2.1)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1.spark2 by Apache Hive
0: jdbc:hive2://192.168.0.206:10000> Drop table if exists local_table2;
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (21.205 seconds)
0: jdbc:hive2://192.168.0.206:10000>
0: jdbc:hive2://192.168.0.206:10000> CREATE TABLE if not exists local_table2( shortField SHORT, intField INT, bigintField LONG, doubleField DOUBLE, stringField STRING, timestampField TIMESTAMP, decimalField DECIMAL(18,2), dateField DATE, charField CHAR(5), floatField FLOAT ) STORED BY 'carbondata' LOCATION 's3a://carbonstore/local_table2' TBLPROPERTIES('SORT_COLUMNS'='');
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (1.296 seconds)
0: jdbc:hive2://192.168.0.206:10000>
0: jdbc:hive2://192.168.0.206:10000>
0: jdbc:hive2://192.168.0.206:10000> LOAD DATA LOCAL INPATH '/huawei/xubo/git/carbondata2/examples/spark2/src/main/resources/data1.csv' INTO TABLE local_table2 OPTIONS('HEADER'='true');
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (3.339 seconds)
0: jdbc:hive2://192.168.0.206:10000>
0: jdbc:hive2://192.168.0.206:10000> select * from local_table2;
+-------------+-----------+--------------+--------------+--------------+------------------------+---------------+-------------+------------+-------------+--+
| shortfield | intfield | bigintfield | doublefield | stringfield | timestampfield | decimalfield | datefield | charfield | floatfield |
+-------------+-----------+--------------+--------------+--------------+------------------------+---------------+-------------+------------+-------------+--+
| 1 | 10 | 1100 | 48.4 | spark | 2015-04-23 12:01:01.0 | 1.23 | 2015-04-23 | aaa | 2.5 |
| 5 | 17 | 1140 | 43.4 | spark | 2015-07-27 12:01:02.0 | 3.45 | 2015-07-27 | bbb | 2.5 |
| 1 | 11 | 1100 | 44.4 | flink | 2015-05-23 12:01:03.0 | 23.23 | 2015-05-23 | ccc | 2.5 |
| 1 | 10 | 1150 | 43.4 | spark | 2015-07-24 12:01:04.0 | 254.12 | 2015-07-24 | ddd | 2.5 |
| 1 | 10 | 1100 | 47.4 | spark | 2015-07-23 12:01:05.0 | 876.14 | 2015-07-23 | eeee | 3.5 |
| 3 | 14 | 1160 | 43.4 | hive | 2015-07-26 12:01:06.0 | 3454.32 | 2015-07-26 | ff | 2.5 |
| 2 | 10 | 1100 | 43.4 | impala | 2015-07-23 12:01:07.0 | 456.98 | 2015-07-23 | ggg | 2.5 |
| 1 | 10 | 1100 | 43.4 | spark | 2015-05-23 12:01:08.0 | 32.53 | 2015-05-23 | hhh | 2.5 |
| 4 | 16 | 1130 | 42.4 | impala | 2015-07-23 12:01:09.0 | 67.23 | 2015-07-23 | iii | 2.5 |
| 1 | 10 | 1100 | 43.4 | spark | 2015-07-23 12:01:10.0 | 832.23 | 2015-07-23 | jjj | 2.5 |
+-------------+-----------+--------------+--------------+--------------+------------------------+---------------+-------------+------------+-------------+--+
10 rows selected (0.959 seconds)
0: jdbc:hive2://192.168.0.206:10000>
```
- [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull
https://github.com/xubo245/carbondata CARBONDATA-2260-carbonthriftserver-support-s3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2073.patchTo close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2073
----
commit c59fe8d687dac498fccf341b972d2a0c5ddadbf5
Author: root <601450868@...>
Date: 2018-03-17T09:45:34Z
[CARBONDATA-2260] CarbonThriftServer should support store carbon table on S3
UPDATE SCALA STYLE
FIX scala style
fix scalastyle
support run this class with local, hdfs and s2 mode
----
---
Locked
