Hi:
I use spark streaming insert data into carbondata. I find that the schema of my data don't match the schema of table. after the method of class RowParserImpl: @Override public Object[] parseRow(Object[] row) { if (row == null) { return new String[numberOfColumns]; } // If number of columns are less in a row then create new array with same size of header. if (row.length < numberOfColumns) { String[] temp = new String[numberOfColumns]; System.arraycopy(row, 0, temp, 0, row.length); row = temp; } Object[] out = new Object[genericParsers.length]; for (int i = 0; i < genericParsers.length; i++) { Object obj = row[inputMapping[i]]; out[outputMapping[i]] = genericParsers[i].parse(obj); } return out; } The schema is "id,name,city,salary".but the table schema is "name,city,id,salary" in the class RowConverterImpl . why? is bug? Best Jocean.shi |
You can get table schema by CarbonTable.getCreateOrderColumn method.
It will return the correct table schema. "name,city,id,salary" is the order of column storage, it is not the table schema. ----- Best Regards David Cai -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Best Regards
David Cai |
I want to using spark streaming insert data into carbondata. But I cann’t find any example for it.can you help for it. Another question is that Spark streaming is support update/delete carbondata table? |
In reply to this post by David CaiQiang
HI:
I dont't use CarbonTable.getCreateOrderColumn method. All the method controller is Carbondata. If my DataFrame schema order don't match table schema order then error Best Jocean.shi David CaiQiang <[hidden email]> 于2019年3月20日周三 下午3:48写道: > You can get table schema by CarbonTable.getCreateOrderColumn method. > It will return the correct table schema. > > "name,city,id,salary" is the order of column storage, it is not the table > schema. > > > > ----- > Best Regards > David Cai > -- > Sent from: > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ > |
HI yanggs:
The example is SparkStreamingExample in source code. Streaming table don't support update/delete Best Jocean.shi Jocean shi <[hidden email]> 于2019年3月20日周三 下午4:09写道: > HI: > > I dont't use CarbonTable.getCreateOrderColumn method. > All the method controller is Carbondata. > If my DataFrame schema order don't match table schema order then error > > Best > Jocean.shi > > > David CaiQiang <[hidden email]> 于2019年3月20日周三 下午3:48写道: > >> You can get table schema by CarbonTable.getCreateOrderColumn method. >> It will return the correct table schema. >> >> "name,city,id,salary" is the order of column storage, it is not the table >> schema. >> >> >> >> ----- >> Best Regards >> David Cai >> -- >> Sent from: >> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ >> > |
In reply to this post by yanggs
There are all documents(include streaming table) under the following link.
https://github.com/apache/carbondata/tree/master/docs You can find all examples in examples/spark2 module: example 1 (support Update/Delete) https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/StreamingUsingBatchLoadExample.scala example 2 (not support Update/Delete) https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/StructuredStreamingExample.scala or: https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/StreamingWithRowParserExample.scala ----- Best Regards David Cai -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Best Regards
David Cai |
Free forum by Nabble | Edit this page |