dev
What time is expected to release a patch, we can test
yixu2001
From: yixu2001
Date: 2017-11-07 15:49
To: dev
Subject: Re: Re: Delegation Token can be issued only with kerberos or web authentication" will occur in yarn cluster
dev
hello there is no recent progress
yixu2001
From: yixu2001
Date: 2017-11-06 16:43
To: dev
CC: prnaresh.naresh; 郭海涛
Subject: Re: Re: Delegation Token can be issued only with kerberos or web authentication" will occur in yarn cluster
dev
I have copied core-site.xml to spark2 conf folder, it does not work.
I will share you the jar I am using(please get the jar file from link
download link ：
https://pan.baidu.com/s/1b6AG5S password：rww2
whould you help me to confirm whether the jar is suitable?
If the jar is not suitable, could you please send me a suitable one to try?
yixu2001
From: Naresh P R
Date: 2017-11-04 01:26
To: dev
Subject: Re: Re: Delegation Token can be issued only with kerberos or web authentication" will occur in yarn cluster
Hi yixu2001,
From hadoop code, i could see IOException("Delegation Token can be issued
only with kerberos or web authentication") is thrown only if authentication
method is set as "SIMPLE".
private boolean isAllowedDelegationTokenOp() throws IOException {
AuthenticationMethod authMethod = this.getConnectionAuthenticationMethod();
return !UserGroupInformation.isSecurityEnabled() || authMethod ==
AuthenticationMethod.KERBEROS || authMethod ==
AuthenticationMethod.KERBEROS_SSL || authMethod ==
AuthenticationMethod.CERTIFICATE;
}
Token<DelegationTokenIdentifier> getDelegationToken(Text renewer)
throws IOException {
....
if (!this.isAllowedDelegationTokenOp()) {
throw new IOException("Delegation Token can be issued only
with kerberos or web authentication");
}
....
}
Can you try to execute queries by copying core-site.xml from hadoop-conf
folder to spark2 conf folder & classpath of spark-submit?
From the provided logs, i could
see carbondata_2.11-1.1.1-bdd-hadoop2.7.2.jar size is 9607344bytes, please
make sure it has only carbondata classes.
I could see Carbon explicitly calling
"TokenCache.obtainTokensForNamenodes" which
is throwing this exception.
If above mentioned steps dint work, you can raise a JIRA to investigate
further on this.
----
Regards,
Naresh P R
On Fri, Nov 3, 2017 at 3:10 PM, yixu2001 <
[hidden email]> wrote:
> prnaresh.naresh, dev
>
> The carbon jar I used does not include hadoop classes & core-site.xml.
> The attachment include the jar list while submtting
> spark job, please confirm it.
> ------------------------------
> yixu2001
>
>
> *From:* Naresh P R <
[hidden email]>
> *Date:* 2017-11-03 16:07
> *To:* yixu2001 <
[hidden email]>
> *Subject:* Re: Re: Delegation Token can be issued only with kerberos or
> web authentication" will occur in yarn cluster
> Hi yixu2001,
>
> Are you using carbon shaded jar with hadoop classes & core-site.xml
> included in carbon jar ?
>
> If so, can you try to use carbondata individual component jars while
> submtting spark job?
>
> As per my understanding, this happens if client core-site.xml has
> hadoop.security.authentication=simple & hdfs is kerberized.
>
> You can also enable verbose to see hadoop jars used in the error trace
> while querying carbon tables.
>
> Also i am not sure whether CarbonData is tested in HDP kerberos cluster.
> ---
> Regards,
> Naresh P R
>
>
> On Fri, Nov 3, 2017 at 8:36 AM, yixu2001 <
[hidden email]> wrote:
>
>> Naresh P R:
>> For the attachments can not be uploaded in maillist，I have
>> add the attachments to the mail for you, please check it.
>> Our platform is installed with HDP 2.4, but spark 2.1 is
>> not included in HDP 2.4, we using spark 2.1 with additiona
>> l installed of apache version.
>> ------------------------------
>> yixu2001
>>
>>
>> *From:* Naresh P R <
[hidden email]>
>> *Date:* 2017-11-02 22:02
>> *To:* dev <
[hidden email]>
>> *Subject:* Re: Re: Delegation Token can be issued only with kerberos or
>> web authentication" will occur in yarn cluster
>> Hi yixu,
>>
>> I am not able to see any attachment in your previous mail.
>> ---
>> Regards,
>> Naresh P R
>>
>> On Thu, Nov 2, 2017 at 4:40 PM, yixu2001 <
[hidden email]> wrote:
>>
>>> dev
>>> Please refer to the attachment "cluster carbon error2.txt"
>>> for the log trace.
>>> In this log, I try 2 query statements:
>>> select * from e_carbon.prod_inst_his prod_inst_his is
>>> a hive table, it success.
>>> select * from e_carbon.prod_inst_his_c prod_inst_his_c i
>>> s a carbon table, it failed.
>>>
>>> I pass the principal in my start script, please refer to the
>>> attachment "testCluster.sh
>>>
>>> ".
>>>
>>> I have set hive.server2.enable.doAs = false in the above tes
>>> t and I have printed it in the log.
>>> ------------------------------
>>> yixu2001
>>>
>>>
>>> *From:* Naresh P R <
[hidden email]>
>>> *Date:* 2017-11-01 19:40
>>> *To:* dev <
[hidden email]>
>>> *Subject:* Re: Delegation Token can be issued only with kerberos or web
>>> authentication" will occur in yarn cluster
>>> Hi,
>>>
>>> Ideally kerberos authentication should work with carbon table, Can you
>>> share us log trace to analyze further more?
>>>
>>> how are you passing the principal in yarn cluster ?
>>>
>>> can you try to set hive.server2.enable.doAs = false & run query on carbon
>>> table ?
>>> ----
>>> Regards,
>>> Naresh P R
>>>
>>> On Wed, Nov 1, 2017 at 3:33 PM, yixu2001 <
[hidden email]> wrote:
>>>
>>> > dev
>>> > I submit a spark application in mode yarn cluster to a cluster with
>>> > kerberos. In this application, it will successfully query a hive
>>> table, but
>>> > when it try to query a carbon table, it failed with infomation
>>> "Delegation
>>> > Token can be issued only with kerberos or web authentication".
>>> >
>>> > If I submit this application in mode yarn client, both hive table and
>>> > carbon table will both success.
>>> >
>>> > And If I submit this application in mode yarn cluster on another
>>> cluster
>>> > without kerberos, both hive table and carbon table will both success.
>>> >
>>> >
>>> > yixu2001
>>> >
>>>
>>>
>>
>
Locked
testCluster.sh