Apache CarbonData Dev Mailing List archive › Apache CarbonData JIRA issues

[GitHub] carbondata pull request #1959: [CARBONDATA-2147][CARBONDATA-2148]Add new Row...

Classic

List

Threaded

61 messages Options

1234

qiuchenjian-2

[GitHub] carbondata pull request #1959: [CARBONDATA-2147][CARBONDATA-2148]Add new Row...

GitHub user zzcclp opened a pull request:

https://github.com/apache/carbondata/pull/1959

[CARBONDATA-2147][CARBONDATA-2148]Add new Row parser to replace current default parser: CSVStreamParserImp

Currently the default value of 'carbon.stream.parser' is CSVStreamParserImp, it transforms InternalRow(0) to Array[Object], InternalRow(0) represents the value of one line which is received from Socket. When it receives data from Kafka, the schema of InternalRow is changed, either it need to assemble the fields of kafka data Row into a String and stored it as InternalRow(0), or define a new parser to convert kafka data Row to Array[Object]. It needs the same operation for every table.

Solution:
Use a new parser called RowStreamParserImpl as the default parser instead of CSVStreamParserImpl, this new parser will automatically convert InternalRow to Array[Object] according to the schema. In general, we will transform source data to a structed Row object, using this way, we do not need to define a parser for every table.

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

- [ ] Any interfaces changed?

- [ ] Any backward compatibility impacted?

- [ ] Document update required?

- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zzcclp/carbondata CARBONDATA-2148

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1959.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1959

----
commit f01de97fff6eb9f326c53f7a4d0be49d378c77c5
Author: Zhang Zhichao <441586683@...>
Date: 2018-02-09T06:49:58Z

[CARBONDATA-2147][CARBONDATA-2148]Add new Row parser to replace current default parser:CSVStreamParserImp

Currently the default value of 'carbon.stream.parser' is CSVStreamParserImp, it transforms InternalRow(0) to Array[Object], InternalRow(0) represents the value of one line which is received from Socket. When it receives data from Kafka, the schema of InternalRow is changed, either it need to assemble the fields of kafka data Row into a String and stored it as InternalRow(0), or define a new parser to convert kafka data Row to Array[Object]. It needs the same operation for every table.

Solution:
Use a new parser called RowStreamParserImpl as the default parser instead of CSVStreamParserImpl, this new parser will automatically convert InternalRow to Array[Object] according to the schema. In general, we will transform source data to a structed Row object, using this way, we do not need to define a parser for every table.

----

---

qiuchenjian-2