[jira] [Commented] (CARBONDATA-2928) query failed when doing merge index during load

[Akash R Nilugal (Jira)]

    [ https://issues.apache.org/jira/browse/CARBONDATA-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16633208#comment-16633208 ]

jiangmanhua commented on CARBONDATA-2928:
-----------------------------------------

Hi, the error message said it can not found the carbondata file, so it should have already read the carbon index/merge index file and the pruning results told us to read the carbondata file. 

 

I am confusing the value `9965100033100001` in `part-0-9965100033100001_batchno0-0-9865-1536751682754.carbondata`. Please check the file.

 

More check I done:

1. let the thread sleep before starting to merge carbon index and query is OK ;

2. While writing merge index, carbon uses `AtoicFileOperations` which first write to a temp file and then rename it.

3. Under `<TablePath>/Metadata/segment/`, you can see `.segment` file which records the index file. After the merge index is written, this file will be updated. So carbon knows to load which index file(s)

> query failed when doing merge index during load
> -----------------------------------------------
>
>                 Key: CARBONDATA-2928
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-2928
>             Project: CarbonData
>          Issue Type: Bug
>          Components: data-load
>    Affects Versions: 1.4.1
>            Reporter: ocean
>            Priority: Major
>             Fix For: NONE
>
>
> In carbondata version 1.4.1, carbonindex file be merged in every load. But when query through thriftserver(about 10QPS), if merge index is in progress, An error will occurs.
>  
> java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException: File does not exist: /warehouse/spark/ae_event_cb_40e_std/productid=534/part-0-9965100033100001_batchno0-0-9865-1536751682754.carbondata
>  at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
>  at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
>  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1828)
>  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799)
>  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712)
>  at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:587)
>  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
>  at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>  at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> at org.apache.hadoop.ipc.Client.call(Client.java:1475)
>  at org.apache.hadoop.ipc.Client.call(Client.java:1412)
>  

--
This message was sent by Atlassian JIRA
(v7.6.3#76005)