[jira] [Created] (CARBONDATA-660) Bad Records Logs and Raw CSVs should get display under segment id instead of Tasks id

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Created] (CARBONDATA-660) Bad Records Logs and Raw CSVs should get display under segment id instead of Tasks id

Akash R Nilugal (Jira)
Priyal Sachdeva created CARBONDATA-660:
------------------------------------------

             Summary: Bad Records Logs and Raw CSVs should get display under segment id instead of Tasks id
                 Key: CARBONDATA-660
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-660
             Project: CarbonData
          Issue Type: Improvement
          Components: data-load
            Reporter: Priyal Sachdeva
            Priority: Minor


create table if not exists Badrecords_test (imei string,AMSize int) STORED BY 'org.apache.carbondata.format';

 LOAD DATA INPATH 'hdfs://hacluster/CSVs/bad_records.csv' into table Badrecords_test OPTIONS('DELIMITER'=',' , 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 'BAD_RECORDS_ACTION'='REDIRECT','FILEHEADER'='imei,AMSize');


Bad Records Logs and raw csvs are getting display under Task ID


linux-61:/srv/OSCON/BigData/HACluster/install/hadoop/datanode #
bin/hadoop fs -ls /tmp/carbon/default/badrecords_test

drwxr-xr-x   - root users          0 2017-01-18 21:08 /tmp/carbon/default/badrecords_test/0--------------------------->Task ID


0: jdbc:hive2://172.168.100.205:23040> show segments for table Badrecords_test;
+--------------------+------------------+--------------------------+--------------------------+--+
| SegmentSequenceId  |      Status      |     Load Start Time      |      Load End Time       |
+--------------------+------------------+--------------------------+--------------------------+--+
| 8                  | Partial Success  | 2017-01-18 21:12:58.018  | 2017-01-18 21:12:59.652  |
| 7                  | Partial Success  | 2017-01-18 21:08:07.426  | 2017-01-18 21:08:11.791  |
| 6                  | Partial Success  | 2017-01-18 21:07:07.645  | 2017-01-18 21:07:08.747  |
| 5                  | Partial Success  | 2017-01-18 19:34:16.163  | 2017-01-18 19:34:18.163  |
| 4                  | Partial Success  | 2017-01-18 19:34:13.669  | 2017-01-18 19:34:15.811  |
| 3                  | Partial Success  | 2017-01-18 19:30:18.753  | 2017-01-18 19:30:19.644  |
| 2                  | Partial Success  | 2017-01-18 19:30:13.508  | 2017-01-18 19:30:15.578  |
| 1                  | Partial Success  | 2017-01-18 19:18:54.787  | 2017-01-18 19:18:54.94   |
| 0                  | Partial Success  | 2017-01-18 19:18:53.741  | 2017-01-18 19:18:54.614  |
+--------------------+------------------+--------------------------+--------------------------+--+

Bad Records Logs and raw csvs are getting display under Task ID. It would be good to have the information of bad records as per the load i.e under segment id..



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)