[jira] [Created] (CARBONDATA-2397) Error while fetching data from a table with complex data type(array_of_struct)

[Akash R Nilugal (Jira)]

Vandana Yadav created CARBONDATA-2397:
-----------------------------------------

             Summary: Error while fetching data from a table with complex data type(array_of_struct)
                 Key: CARBONDATA-2397
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-2397
             Project: CarbonData
          Issue Type: Bug
          Components: data-query
    Affects Versions: 1.4.0
         Environment: spark 2.1,spark 2.2
            Reporter: Vandana Yadav
         Attachments: arrayofstruct.csv

Error while fetching data from a table with the complex data type(array_of_struct)



Steps to reproduce:

Create Table :

create table ARRAY_OF_STRUCT_com (CUST_ID string, YEAR int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, ARRAY_OF_STRUCT array<struct<ID:int,COUNTRY:string,STATE:string,CITI:string,CHECK_DATE:timestamp>>,CARD_COUNT int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) STORED BY 'org.apache.carbondata.format'

Load Data into the table:

LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/complex/arrayofstruct.csv' INTO table ARRAY_OF_STRUCT_com options ('DELIMITER'=',', 'QUOTECHAR'='"', 'FILEHEADER'='CUST_ID,YEAR,MONTH,AGE,GENDER,EDUCATED,IS_MARRIED,ARRAY_OF_STRUCT,CARD_COUNT,DEBIT_COUNT,CREDIT_COUNT,DEPOSIT,HQ_DEPOSIT','COMPLEX_DELIMITER_LEVEL_1'='$','COMPLEX_DELIMITER_LEVEL_2'='&')

 

Execute Query:

select * from array_of_struct_com;

Expected Result: it should display all the data from the table.

Actual Result:

Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 1.0 failed 1 times, most recent failure: Lost task 1.0 in stage 1.0 (TID 2, localhost, executor driver): java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.spark.sql.catalyst.util.ArrayData
 at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.getArray(rows.scala:48)
 at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getArray(rows.scala:194)
 at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
 at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
 at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
 at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)
 at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
 at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
 at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
 at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
 at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
 at org.apache.spark.scheduler.Task.run(Task.scala:108)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

Driver stacktrace: (state=,code=0)

Error log:

18/04/25 11:13:58 INFO SparkExecuteStatementOperation: Running query 'select * from array_of_struct_com' with 4a55f43f-96c7-46c0-9a71-cc66e5dfa641
18/04/25 11:13:58 INFO CarbonSparkSqlParser: Parsing command: select * from array_of_struct_com
18/04/25 11:13:58 INFO HiveMetaStore: 7: get_table : db=bug tbl=array_of_struct_com
18/04/25 11:13:58 INFO audit: ugi=knoldus ip=unknown-ip-addr cmd=get_table : db=bug tbl=array_of_struct_com
18/04/25 11:13:58 INFO HiveMetaStore: 7: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
18/04/25 11:13:58 INFO ObjectStore: ObjectStore, initialize called
18/04/25 11:13:58 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
18/04/25 11:13:58 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
18/04/25 11:13:58 INFO ObjectStore: Initialized ObjectStore
18/04/25 11:13:58 INFO CatalystSqlParser: Parsing command: array<string>
18/04/25 11:13:58 INFO CarbonLRUCache: pool-23-thread-6 Removed entry from InMemory lru cache :: hdfs://localhost:54310/opt/CarbonStore/bug/array_of_struct_com/Fact/Part0/Segment_0/0_batchno0-0-1524572335034.carbonindex
18/04/25 11:13:58 INFO CarbonLRUCache: pool-23-thread-6 Removed entry from InMemory lru cache :: hdfs://localhost:54310/opt/CarbonStore/bug/array_of_struct_com/Fact/Part0/Segment_1/0_batchno0-0-1524575558281.carbonindex
18/04/25 11:13:58 INFO HiveMetaStore: 7: get_table : db=bug tbl=array_of_struct_com
18/04/25 11:13:58 INFO audit: ugi=knoldus ip=unknown-ip-addr cmd=get_table : db=bug tbl=array_of_struct_com
18/04/25 11:13:58 INFO CatalystSqlParser: Parsing command: array<string>
18/04/25 11:13:58 INFO HiveMetaStore: 7: get_database: bug
18/04/25 11:13:58 INFO audit: ugi=knoldus ip=unknown-ip-addr cmd=get_database: bug
18/04/25 11:13:58 INFO HiveMetaStore: 7: get_database: bug
18/04/25 11:13:58 INFO audit: ugi=knoldus ip=unknown-ip-addr cmd=get_database: bug
18/04/25 11:13:58 INFO HiveMetaStore: 7: get_tables: db=bug pat=*
18/04/25 11:13:58 INFO audit: ugi=knoldus ip=unknown-ip-addr cmd=get_tables: db=bug pat=*
18/04/25 11:13:58 INFO TableInfo: pool-23-thread-6 Table block size not specified for bug_array_of_struct_com. Therefore considering the default value 1024 MB
18/04/25 11:13:58 INFO CarbonLateDecodeRule: pool-23-thread-6 skip CarbonOptimizer
18/04/25 11:13:58 INFO CarbonLateDecodeRule: pool-23-thread-6 Skip CarbonOptimizer
18/04/25 11:13:58 INFO CodeGenerator: Code generated in 63.208107 ms
18/04/25 11:13:58 INFO TableInfo: pool-23-thread-6 Table block size not specified for bug_array_of_struct_com. Therefore considering the default value 1024 MB
18/04/25 11:13:58 INFO BlockletDataMap: pool-23-thread-6 Time taken to load blocklet datamap from file : hdfs://localhost:54310/opt/CarbonStore/bug/array_of_struct_com/Fact/Part0/Segment_0/0_batchno0-0-1524572335034.carbonindexis 2
18/04/25 11:13:58 INFO BlockletDataMap: pool-23-thread-6 Time taken to load blocklet datamap from file : hdfs://localhost:54310/opt/CarbonStore/bug/array_of_struct_com/Fact/Part0/Segment_1/0_batchno0-0-1524575558281.carbonindexis 3
18/04/25 11:13:58 INFO CarbonScanRDD:
 Identified no.of.blocks: 2,
 no.of.tasks: 2,
 no.of.nodes: 0,
 parallelism: 4
 
18/04/25 11:13:58 INFO SparkContext: Starting job: run at AccessController.java:0
18/04/25 11:13:58 INFO DAGScheduler: Got job 2 (run at AccessController.java:0) with 2 output partitions
18/04/25 11:13:58 INFO DAGScheduler: Final stage: ResultStage 2 (run at AccessController.java:0)
18/04/25 11:13:58 INFO DAGScheduler: Parents of final stage: List()
18/04/25 11:13:58 INFO DAGScheduler: Missing parents: List()
18/04/25 11:13:58 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[8] at run at AccessController.java:0), which has no missing parents
18/04/25 11:13:58 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 35.7 KB, free 366.2 MB)
18/04/25 11:13:58 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 27.7 KB, free 366.2 MB)
18/04/25 11:13:58 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.2.102:40679 (size: 27.7 KB, free: 366.2 MB)
18/04/25 11:13:58 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006
18/04/25 11:13:58 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 2 (MapPartitionsRDD[8] at run at AccessController.java:0) (first 15 tasks are for partitions Vector(0, 1))
18/04/25 11:13:58 INFO TaskSchedulerImpl: Adding task set 2.0 with 2 tasks
18/04/25 11:13:58 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 3, localhost, executor driver, partition 0, ANY, 6524 bytes)
18/04/25 11:13:58 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 4, localhost, executor driver, partition 1, ANY, 6534 bytes)
18/04/25 11:13:58 INFO Executor: Running task 0.0 in stage 2.0 (TID 3)
18/04/25 11:13:58 INFO Executor: Running task 1.0 in stage 2.0 (TID 4)
18/04/25 11:13:58 INFO TableInfo: Executor task launch worker for task 3 Table block size not specified for bug_array_of_struct_com. Therefore considering the default value 1024 MB
18/04/25 11:13:58 INFO TableInfo: Executor task launch worker for task 4 Table block size not specified for bug_array_of_struct_com. Therefore considering the default value 1024 MB
18/04/25 11:13:58 INFO AbstractQueryExecutor: [Executor task launch worker for task 3][partitionID:com;queryID:18146683300601] Query will be executed on table: array_of_struct_com
18/04/25 11:13:58 INFO AbstractQueryExecutor: [Executor task launch worker for task 4][partitionID:com;queryID:18146683300601] Query will be executed on table: array_of_struct_com
18/04/25 11:13:58 INFO ResultCollectorFactory: [Executor task launch worker for task 3][partitionID:com;queryID:18146683300601] Row based dictionary collector is used to scan and collect the data
18/04/25 11:13:58 INFO ResultCollectorFactory: [Executor task launch worker for task 4][partitionID:com;queryID:18146683300601] Restructure based dictionary collector is used to scan and collect the data
18/04/25 11:13:58 INFO UnsafeMemoryManager: [Executor task launch worker for task 4][partitionID:com;queryID:18146683300601] Total memory used after task 18146962815012 is 6581 Current tasks running now are : [17522539140626, 17607895858118, 18146866243005]
18/04/25 11:13:58 ERROR Executor: Exception in task 1.0 in stage 2.0 (TID 4)
java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.spark.sql.catalyst.util.ArrayData
 at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.getArray(rows.scala:48)
 at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getArray(rows.scala:194)
 at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
 at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
 at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
 at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)
 at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
 at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
 at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
 at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
 at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
 at org.apache.spark.scheduler.Task.run(Task.scala:108)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)