spark streaming insert data error

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

spark streaming insert data error

Jocean shi
Hi:

I use spark streaming insert data into carbondata. I find that the schema
of my data don't match the schema of table.
after the method of class  RowParserImpl:

@Override
public Object[] parseRow(Object[] row) {
  if (row == null) {
    return new String[numberOfColumns];
  }
  // If number of columns are less in a row then create new array with
same size of header.
  if (row.length < numberOfColumns) {
    String[] temp = new String[numberOfColumns];
    System.arraycopy(row, 0, temp, 0, row.length);
    row = temp;
  }
  Object[] out = new Object[genericParsers.length];
  for (int i = 0; i < genericParsers.length; i++) {
    Object obj = row[inputMapping[i]];
    out[outputMapping[i]] = genericParsers[i].parse(obj);
  }
  return out;
}

The schema is "id,name,city,salary".but the table schema is
"name,city,id,salary" in

the class RowConverterImpl .

why? is bug?


Best

Jocean.shi
Reply | Threaded
Open this post in threaded view
|

Re: spark streaming insert data error

David CaiQiang
You can get table schema by CarbonTable.getCreateOrderColumn method.
It will return the correct table schema.

"name,city,id,salary" is the order of column storage, it is not the table
schema.



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Best Regards
David Cai
Reply | Threaded
Open this post in threaded view
|

答复: spark streaming insert data error

yanggs

I want to using spark streaming insert data into carbondata.
But I cann’t find any example for it.can you help for it.
Another question is that Spark streaming is support update/delete carbondata table?



Reply | Threaded
Open this post in threaded view
|

Re: spark streaming insert data error

Jocean shi
In reply to this post by David CaiQiang
HI:

I dont't use CarbonTable.getCreateOrderColumn method.
All the method controller is Carbondata.
If my DataFrame schema order don't match table schema order then error

Best
Jocean.shi


David CaiQiang <[hidden email]> 于2019年3月20日周三 下午3:48写道:

> You can get table schema by CarbonTable.getCreateOrderColumn method.
> It will return the correct table schema.
>
> "name,city,id,salary" is the order of column storage, it is not the table
> schema.
>
>
>
> -----
> Best Regards
> David Cai
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>
Reply | Threaded
Open this post in threaded view
|

Re: spark streaming insert data error

Jocean shi
HI yanggs:

The example is SparkStreamingExample in source code. Streaming table don't
support update/delete

Best
Jocean.shi

Jocean shi <[hidden email]> 于2019年3月20日周三 下午4:09写道:

> HI:
>
> I dont't use CarbonTable.getCreateOrderColumn method.
> All the method controller is Carbondata.
> If my DataFrame schema order don't match table schema order then error
>
> Best
> Jocean.shi
>
>
> David CaiQiang <[hidden email]> 于2019年3月20日周三 下午3:48写道:
>
>> You can get table schema by CarbonTable.getCreateOrderColumn method.
>> It will return the correct table schema.
>>
>> "name,city,id,salary" is the order of column storage, it is not the table
>> schema.
>>
>>
>>
>> -----
>> Best Regards
>> David Cai
>> --
>> Sent from:
>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: 答复: spark streaming insert data error

David CaiQiang
In reply to this post by yanggs