[jira] [Commented] (CARBONDATA-4021) With Index server running, Upon executing count* we are getting the below error, after adding the parquet and ORC segment.

[Akash R Nilugal (Jira)]

    [ https://issues.apache.org/jira/browse/CARBONDATA-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237543#comment-17237543 ]

Karan commented on CARBONDATA-4021:
-----------------------------------

Caching parquet or ORC segments in Index Server is not supported. Please do not enable index server while querying on carbonn data table having parquet or ORC segments. Even if index Server is ON, please make sure that fallback is not disabled.

> With Index server running, Upon executing count* we are getting the below error, after adding the parquet and ORC segment.
> ---------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-4021
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-4021
>             Project: CarbonData
>          Issue Type: Bug
>    Affects Versions: 2.0.0
>            Reporter: Prasanna Ravichandran
>            Priority: Major
>
> We are getting below issues while index server enable and index server fallback disable is configured as true. With count* we are getting the below error, after adding the parquet and ORC segment.
> Queries and error:
> > use rps;
> +---------+|
> Result  |
> +---------+
> +---------+
> No rows selected (0.054 seconds)
> > drop table if exists uniqdata;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.229 seconds)
> > CREATE TABLE uniqdata (cust_id int,cust_name String,active_emui_version string, dob timestamp, doj timestamp, bigint_column1 bigint,bigint_column2 bigint,decimal_column1 decimal(30,10), decimal_column2 decimal(36,36),double_column1 double, double_column2 double,integer_column1 int) stored as carbondata;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.756 seconds)
> > load data inpath 'hdfs://hacluster/user/prasanna/2000_UniqData.csv' into table uniqdata options('fileheader'='cust_id,cust_name,active_emui_version,dob,doj,bigint_column1,bigint_column2,decimal_column1,decimal_column2,double_column1,double_column2,integer_column1','bad_records_action'='force');
> INFO  : Execution ID: 95
> +---------+|
> Result  |
> +---------+
> +---------+
> No rows selected(2.789 seconds)
>  > use default;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.052 seconds)
>  > drop table if exists uniqdata;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (1.122 seconds)
> > CREATE TABLE uniqdata (cust_id int,cust_name String,active_emui_version string, dob timestamp, doj timestamp, bigint_column1 bigint,bigint_column2 bigint,decimal_column1 decimal(30,10), decimal_column2 decimal(36,36),double_column1 double, double_column2 double,integer_column1 int) stored as carbondata;
> +---------+
> |  Result  |
> +---------+
> +---------+
> No rows selected (0.508 seconds)
> > load data inpath 'hdfs://hacluster/user/prasanna/2000_UniqData.csv' into table uniqdata options('fileheader'='cust_id,cust_name,active_emui_version,dob,doj,bigint_column1,bigint_column2,decimal_column1,decimal_column2,double_column1,double_column2,integer_column1','bad_records_action'='force');
> INFO  : Execution ID: 108
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (1.316 seconds)
> > drop table if exists uniqdata_parquet;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.668 seconds)
> > CREATE TABLE uniqdata_parquet (cust_id int,cust_name String,active_emui_version string, dob timestamp, doj timestamp, bigint_column1 bigint,bigint_column2 bigint,decimal_column1 decimal(30,10), decimal_column2 decimal(36,36),double_column1 double, double_column2 double,integer_column1 int) stored as parquet;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.397 seconds)
> > insert into uniqdata_parquet select * from uniqdata;
> INFO  : Execution ID: 116
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (4.805 seconds)
> >  drop table if exists uniqdata_orc;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.553 seconds)
> > CREATE TABLE uniqdata_orc (cust_id int,cust_name String,active_emui_version string, dob timestamp, doj timestamp, bigint_column1 bigint,bigint_column2 bigint,decimal_column1 decimal(30,10), decimal_column2 decimal(36,36),double_column1 double, double_column2 double,integer_column1 int) using orc;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.396 seconds)
> > insert into uniqdata_orc select * from uniqdata;
> INFO  : Execution ID: 122
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (3.403 seconds)
> > use rps;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.06 seconds)
> > Alter table uniqdata add segment options ('path'='hdfs://hacluster/user/hive/warehouse/uniqdata_parquet','format'='parquet');
> INFO  : Execution ID: 126
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (1.511 seconds)
> > Alter table uniqdata add segment options ('path'='hdfs://hacluster/user/hive/warehouse/uniqdata_orc','format'='orc');
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.716 seconds)
> > select count(*) from uniqdata;
> Error: java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.io.IOException): java.security.PrivilegedActionException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 54.0 failed 4 times, most recent failure: Lost task 2.3 in stage 54.0 (TID 118, linux-16, executor 2): java.io.IOException: Problem in loading segment blocks: java.io.IOException: org.apache.thrift.protocol.TProtocolException: Required field 'version' was not found in serialized data! Struct: org.apache.carbondata.format.IndexHeader$IndexHeaderStandardScheme@30eec3e5 at org.apache.carbondata.core.indexstore.BlockletIndexStore.getAll(BlockletIndexStore.java:201) at org.apache.carbondata.core.indexstore.blockletindex.BlockletIndexFactory.getIndexes(BlockletIndexFactory.java:181) at org.apache.carbondata.core.indexstore.blockletindex.BlockletIndexFactory.getIndexes(BlockletIndexFactory.java:141) at org.apache.carbondata.core.index.TableIndex.prune(TableIndex.java:131) at org.apache.carbondata.core.index.IndexInputFormat$1.initialize(IndexInputFormat.java:174) at org.apache.carbondata.indexserver.DistributedPruneRDD$$anonfun$org$apache$carbondata$indexserver$DistributedPruneRDD$$generateFuture$1$$anonfun$apply$1.apply(DistributedPruneRDD.scala:141) at org.apache.carbondata.indexserver.DistributedPruneRDD$$anonfun$org$apache$carbondata$indexserver$DistributedPruneRDD$$generateFuture$1$$anonfun$apply$1.apply(DistributedPruneRDD.scala:138) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.immutable.List.foreach(List.scala:392) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at scala.collection.immutable.List.flatMap(List.scala:355) at org.apache.carbondata.indexserver.DistributedPruneRDD$$anonfun$org$apache$carbondata$indexserver$DistributedPruneRDD$$generateFuture$1.apply(DistributedPruneRDD.scala:138) at org.apache.carbondata.indexserver.DistributedPruneRDD$$anonfun$org$apache$carbondata$indexserver$DistributedPruneRDD$$generateFuture$1.apply(DistributedPruneRDD.scala:138) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.RuntimeException: java.io.IOException: org.apache.thrift.protocol.TProtocolException: Required field 'version' was not found in serialized data! Struct: org.apache.carbondata.format.IndexHeader$IndexHeaderStandardScheme@30eec3e5 at org.apache.carbondata.core.indexstore.BlockletIndexStore.get(BlockletIndexStore.java:160) at org.apache.carbondata.core.indexstore.BlockletIndexStore.getAll(BlockletIndexStore.java:190) ... 18 moreCaused by: java.io.IOException: org.apache.thrift.protocol.TProtocolException: Required field 'version' was not found in serialized data! Struct: org.apache.carbondata.format.IndexHeader$IndexHeaderStandardScheme@30eec3e5 at org.apache.carbondata.core.reader.ThriftReader.read(ThriftReader.java:146) at org.apache.carbondata.core.reader.CarbonIndexFileReader.readIndexHeader(CarbonIndexFileReader.java:54) at org.apache.carbondata.core.util.AbstractDataFileFooterConverter.getIndexInfo(AbstractDataFileFooterConverter.java:151) at org.apache.carbondata.core.util.BlockletIndexUtil.getBlockMetaInfoMap(BlockletIndexUtil.java:113) at org.apache.carbondata.core.indexstore.BlockletIndexStore.get(BlockletIndexStore.java:112) ... 19 moreCaused by: org.apache.thrift.protocol.TProtocolException: Required field 'version' was not found in serialized data! Struct: org.apache.carbondata.format.IndexHeader$IndexHeaderStandardScheme@30eec3e5 at org.apache.carbondata.format.IndexHeader$IndexHeaderStandardScheme.read(IndexHeader.java:829) at org.apache.carbondata.format.IndexHeader$IndexHeaderStandardScheme.read(IndexHeader.java:748) at org.apache.carbondata.format.IndexHeader.read(IndexHeader.java:656) at org.apache.carbondata.core.reader.ThriftReader.read(ThriftReader.java:144) ... 23 more
> Driver stacktrace: at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1717) at org.apache.carbondata.indexserver.IndexServer$.doAs(IndexServer.scala:116) at org.apache.carbondata.indexserver.IndexServer$.getSplits(IndexServer.scala:171) at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:550) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:985) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:913) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1737) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2876) (state=,code=0)
>  

--
This message was sent by Atlassian Jira
(v8.3.4#803005)