[jira] [Commented] (CARBONDATA-1421) Auto Compaction Failing in CarbonData Loading

[Akash R Nilugal (Jira)]

    [ https://issues.apache.org/jira/browse/CARBONDATA-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146837#comment-16146837 ]

Zhichao  Zhang commented on CARBONDATA-1421:
--------------------------------------------

Hi Pallavi Singh, I run below code and it worked with pr-1266,

My config:
carbon.enable.auto.load.merge=true
carbon.compaction.level.threshold=2,4

My code:

{code:java}
import spark.implicits._
val df1 = spark.sparkContext.parallelize(0 to 50)
  .map(x => ("a", x.toString(), (x % 2).toString(), x, x.toLong, x * 2))
  .toDF("stringField1", "stringField2", "stringField3", "intField", "longField", "int2Field")
 
val df2 = spark.sparkContext.parallelize(51 to 100)
  .map(x => ("b", x.toString(), (x % 2).toString(), x, x.toLong, x * 2))
  .toDF("stringField1", "stringField2", "stringField3", "intField", "longField", "int2Field")
 
val df3 = df1.union(df2)
spark.sql("DROP TABLE IF EXISTS study_carbondata ").show()
spark.sql("""
    |  CREATE TABLE IF NOT EXISTS study_carbondata (
    |    stringField1          string,
    |    stringField2          string,
    |    stringField3          string,
    |    intField              int,
    |    longField             bigint,
    |    int2Field             int
    |  )
    |  STORED BY 'carbondata'
    |  TBLPROPERTIES(
    |    'TABLE_BLOCKSIZE'='8'
    |  )
   """.stripMargin)
   df3.write
  .format("carbondata")
  .option("tableName", "study_carbondata")
  .option("compress", "true")  // just valid when tempCSV is true
  .option("tempCSV", "false")
  .option("single_pass", "true")
  .option("sort_scope", "LOCAL_SORT") //GLOBAL_SORT  LOCAL_SORT
  .mode(SaveMode.Append)
  .save()
df3.write
  .format("carbondata")
  .option("tableName", "study_carbondata")
  .option("compress", "true")  // just valid when tempCSV is true
  .option("tempCSV", "false")
  .option("single_pass", "true")
  .option("sort_scope", "LOCAL_SORT") //GLOBAL_SORT  LOCAL_SORT
  .mode(SaveMode.Append)
  .save()
 
{code}

> Auto Compaction Failing in CarbonData Loading
> ---------------------------------------------
>
>                 Key: CARBONDATA-1421
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-1421
>             Project: CarbonData
>          Issue Type: Bug
>          Components: data-load
>    Affects Versions: 1.2.0
>            Reporter: Pallavi Singh
>             Fix For: 1.2.0
>
>
> I ran the create query followed by multiple load queries and the auto-compaction is failing.
> 0: jdbc:hive2://localhost:10000> LOAD DATA inpath 'hdfs://localhost:54310/data/4000_UniqData.csv' INTO table uniqdata options('DELIMITER'=',', 'FILEHEADER'='CUST_ID, CUST_NAME, ACTIVE_EMUI_VERSION, DOB, DOJ, BIGINT_COLUMN1, BIGINT_COLUMN2, DECIMAL_COLUMN1, DECIMAL_COLUMN2, Double_COLUMN1, Double_COLUMN2, INTEGER_COLUMN1');
> +---------+--+
> | Result  |
> +---------+--+
> +---------+--+
> No rows selected (1.183 seconds)
> 0: jdbc:hive2://localhost:10000> LOAD DATA inpath 'hdfs://localhost:54310/data/5000_UniqData.csv' INTO table uniqdata options('DELIMITER'=',', 'FILEHEADER'='CUST_ID, CUST_NAME, ACTIVE_EMUI_VERSION, DOB, DOJ, BIGINT_COLUMN1, BIGINT_COLUMN2, DECIMAL_COLUMN1, DECIMAL_COLUMN2, Double_COLUMN1, Double_COLUMN2, INTEGER_COLUMN1');
> Error: java.lang.Exception: Dataload is success. Auto-Compaction has failed. Please check logs. (state=,code=0)
> 0: jdbc:hive2://localhost:10000> LOAD DATA inpath 'hdfs://localhost:54310/data/7000_UniqData.csv' INTO table uniqdata options('DELIMITER'=',', 'FILEHEADER'='CUST_ID, CUST_NAME, ACTIVE_EMUI_VERSION, DOB, DOJ, BIGINT_COLUMN1, BIGINT_COLUMN2, DECIMAL_COLUMN1, DECIMAL_COLUMN2, Double_COLUMN1, Double_COLUMN2, INTEGER_COLUMN1');
> Error: java.lang.Exception: Dataload is success. Auto-Compaction has failed. Please check logs. (state=,code=0)
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000> show segments for table uniqdata;
> +--------------------+----------+--------------------------+--------------------------+--+
> | SegmentSequenceId  |  Status  |     Load Start Time      |      Load End Time       |
> +--------------------+----------+--------------------------+--------------------------+--+
> | 4                  | Success  | 2017-08-29 10:37:13.053  | 2017-08-29 10:37:13.888  |
> | 3                  | Success  | 2017-08-29 10:36:57.851  | 2017-08-29 10:36:59.08   |
> | 2                  | Success  | 2017-08-29 10:36:49.439  | 2017-08-29 10:36:50.373  |
> | 1                  | Success  | 2017-08-29 10:36:37.365  | 2017-08-29 10:36:38.768  |
> | 0                  | Success  | 2017-08-29 10:36:21.011  | 2017-08-29 10:36:26.1    |
> +--------------------+----------+--------------------------+--------------------------+--+
> 5 rows selected (0.099 seconds)
> 0: jdbc:hive2://localhost:10000> LOAD DATA inpath 'hdfs://localhost:54310/data/7000_UniqData.csv' INTO table uniqdata options('DELIMITER'=',', 'FILEHEADER'='CUST_ID, CUST_NAME, ACTIVE_EMUI_VERSION, DOB, DOJ, BIGINT_COLUMN1, BIGINT_COLUMN2, DECIMAL_COLUMN1, DECIMAL_COLUMN2, Double_COLUMN1, Double_COLUMN2, INTEGER_COLUMN1');
> Error: java.lang.Exception: Dataload is success. Auto-Compaction has failed. Please check logs. (state=,code=0)
> 0: jdbc:hive2://localhost:10000> show segments for table uniqdata;
> +--------------------+----------+--------------------------+--------------------------+--+
> | SegmentSequenceId  |  Status  |     Load Start Time      |      Load End Time       |
> +--------------------+----------+--------------------------+--------------------------+--+
> | 5                  | Success  | 2017-08-29 10:38:15.727  | 2017-08-29 10:38:16.548  |
> | 4                  | Success  | 2017-08-29 10:37:13.053  | 2017-08-29 10:37:13.888  |
> | 3                  | Success  | 2017-08-29 10:36:57.851  | 2017-08-29 10:36:59.08   |
> | 2                  | Success  | 2017-08-29 10:36:49.439  | 2017-08-29 10:36:50.373  |
> | 1                  | Success  | 2017-08-29 10:36:37.365  | 2017-08-29 10:36:38.768  |
> | 0                  | Success  | 2017-08-29 10:36:21.011  | 2017-08-29 10:36:26.1    |
> +--------------------+----------+--------------------------+--------------------------+--+

--
This message was sent by Atlassian JIRA
(v6.4.14#64029)