[jira] [Updated] (CARBONDATA-1029) Load data time difference with Single-pass load.

[Akash R Nilugal (Jira)]

     [ https://issues.apache.org/jira/browse/CARBONDATA-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod Rohilla updated CARBONDATA-1029:
--------------------------------------
    Description:
Load data without single pass takes less time as compare to Single-pass load.

Note :CSV Size is 4.00 GB.

Result:

A) Data Load without Single Pass:

0: jdbc:hive2://localhost:10000> LOAD DATA INPATH 'hdfs://hadoop-master:54310/data/uniqdata_bench14.csv' into table uniqdata OPTIONS('DELIMITER'=',' , 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
+---------+--+
| Result  |
+---------+--+
+---------+--+
No rows selected (114.641 seconds)

B) Load Data with Single Pass:

0: jdbc:hive2://localhost:10000> LOAD DATA INPATH 'hdfs://hadoop-master:54310/data/uniqdata_bench14.csv' into table uniqdata OPTIONS('DELIMITER'=',' , 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true');
+---------+--+
| Result  |
+---------+--+
+---------+--+
No rows selected (123.858 seconds)


Expected Result: If user load data with Single-pass then it should take less time as compare to without single pass load.

  was:
Load data without single pass takes less time as compare to Single-pass load.

Note :CSV Size is 10.21 GB.

Result:

A) Data Load without Single Pass:

0: jdbc:hive2://localhost:10000> LOAD DATA INPATH 'hdfs://hadoop-master:54310/data/uniqdata_bench14.csv' into table uniqdata OPTIONS('DELIMITER'=',' , 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
+---------+--+
| Result  |
+---------+--+
+---------+--+
No rows selected (114.641 seconds)

B) Load Data with Single Pass:

0: jdbc:hive2://localhost:10000> LOAD DATA INPATH 'hdfs://hadoop-master:54310/data/uniqdata_bench14.csv' into table uniqdata OPTIONS('DELIMITER'=',' , 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true');
+---------+--+
| Result  |
+---------+--+
+---------+--+
No rows selected (123.858 seconds)


Expected Result: If user load data with Single-pass then it should take less time as compare to without single pass load.

> Load data time difference with Single-pass load.
> -------------------------------------------------
>
>                 Key: CARBONDATA-1029
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-1029
>             Project: CarbonData
>          Issue Type: Bug
>          Components: data-load
>    Affects Versions: 1.1.0
>         Environment: Spark  2.1, AWS Cluster
>            Reporter: Vinod Rohilla
>            Priority: Minor
>
> Load data without single pass takes less time as compare to Single-pass load.
> Note :CSV Size is 4.00 GB.
> Result:
> A) Data Load without Single Pass:
> 0: jdbc:hive2://localhost:10000> LOAD DATA INPATH 'hdfs://hadoop-master:54310/data/uniqdata_bench14.csv' into table uniqdata OPTIONS('DELIMITER'=',' , 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> +---------+--+
> | Result  |
> +---------+--+
> +---------+--+
> No rows selected (114.641 seconds)
> B) Load Data with Single Pass:
> 0: jdbc:hive2://localhost:10000> LOAD DATA INPATH 'hdfs://hadoop-master:54310/data/uniqdata_bench14.csv' into table uniqdata OPTIONS('DELIMITER'=',' , 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true');
> +---------+--+
> | Result  |
> +---------+--+
> +---------+--+
> No rows selected (123.858 seconds)
> Expected Result: If user load data with Single-pass then it should take less time as compare to without single pass load.

--
This message was sent by Atlassian JIRA
(v6.3.15#6346)