Posted by
xm_zzc on
Oct 17, 2018; 9:51am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/java-lang-NegativeArraySizeException-occurred-when-compact-tp65023p65181.html
Hi Kunal Kapoor:
1. No;
2. query unsuccessfully, I use Carbon SDK Reader to read that wrong
segment and it failed too.
3. the schema of the table:
| rt | string
| timestamp_1min | bigint
| timestamp_5min | bigint
| timestamp_1hour | bigint
| customer_id | bigint
| transport_id | bigint
| transport_code | string
| tcp_udp | int
| pre_hdt_id | string
| hdt_id | string
| status | int
| is_end_user | int
| transport_type | string
| transport_type_nam | string
| fcip | string
| host | string
| cip | string
| code | int
| conn_status | int
| recv | bigint
| send | bigint
| msec | bigint
| dst_prefix | string
| next_type | int
| next | string
| hdt_sid | string
| from_endpoint_type | int
| to_endpoint_type | int
| fcip_view | string
| fcip_country | string
| fcip_province | string
| fcip_city | string
| fcip_longitude | string
| fcip_latitude | string
| fcip_node_name | string
| fcip_node_name_cn | string
| host_view | string
| host_country | string
| host_province | string
| host_city | string
| host_longitude | string
| host_latitude | string
| cip_view | string
| cip_country | string
| cip_province | string
| cip_city | string
| cip_longitude | string
| cip_latitude | string
| cip_node_name | string
| cip_node_name_cn | string
| dtp_send | string
| client_port | int
| server_ip | string
| server_port | int
| state | string
| response_code | int
| access_domain | string
| valid | int
| min_batch_time | bigint
| update_time | bigint
| |
| ##Detailed Table Information |
| Database Name | hdt_sys
| Table Name | transport_access_log
| CARBON Store Path | hdfs://hdtcluster/carbon_store
| Comment |
| Table Block Size | 512 MB
| Table Data Size | 777031634135
| Table Index Size | 72894232
| Last Update Time | 1539769299990
| SORT_SCOPE | local_sort
| Streaming | true
| MAJOR_COMPACTION_SIZE | 4096
| AUTO_LOAD_MERGE | true
| COMPACTION_LEVEL_THRESHOLD | 2,8
| |
| ##Detailed Column property |
| ADAPTIVE |
| SORT_COLUMNS |
is_end_user,status,customer_id,access_domain,transport_id,timestamp_1hour,timestamp_1min,conn_status,pre_hdt_id,tcp_udp,transport_code,fcip,cip
|
I think maybe the data are written wrongly when generated segment because of
the issue : MemoryBlock is cleaned by some other thread and result in
wrong data when write data.
Now I deleted the wrong segment and then patch the PR#2796, and continue to
run stream app. If the exception no longer occur, it proves that PR#2796
works. Right?
--
Sent from:
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/