[已解决] [HiveCatalog]Kerberos GSS initiate failed, No valid credentials provided, Cannot read from Sys

问题说明

部署一个连接Hive的Java应用程序，遇到这个Kerberos报错的问题，查了一天，记录一下

问题现象

• Kerberos GSS initiate failed
• No valid credentials provided (Mechanism level: Attempt to obtain new INITIATE credentials failed! (null))
• Cannot read from System.in

javax.security.sasl.SaslException: GSS initiate failed
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[na:1.8.0_351]
        at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) [hive-exec-1.1.0-cdh5.12.1-slankka.jar:1.1.0-cdh5.12.1]       
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[hive-exec-1.1.0-cdh5.12.1-slankka.jar:1.1.0-cdh5.12.1]
        at .....
        at java.lang.Thread.run(Thread.java:750) ~[na:1.8.0_351]
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new INITIATE credentials failed! (null))
        at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:386) ~[na:1.8.0_351]
        ... 44 common frames omitted
Caused by: javax.security.auth.login.LoginException: Cannot read from System.in
        at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:871) ~[na:1.8.0_351]
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:710) ~[na:1.8.0_351]
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) ~[na:1.8.0_351]

排查过程

打开 Kerberos debug:

-Dsun.security.krb5.debug=true

关键信息

>>>KinitOptions cache name is /opt/userdata/krb5cache/0/krb5cc_0.jSzu-aKO
>>>KinitOptions cache name is /opt/userdata/krb5cache/0/krb5cc_0.jSzu-aKO
>>>KinitOptions cache name is /opt/userdata/krb5cache/0/krb5cc_0.jSzu-aKO
>>>KinitOptions cache name is /opt/userdata/krb5cache/0/krb5cc_0.jSzu-aKO
>>>KinitOptions cache name is /opt/userdata/krb5cache/0/krb5cc_0.jSzu-aKO
>>>KinitOptions cache name is /opt/userdata/krb5cache/0/krb5cc_0.jSzu-aKO

网络搜索

https://bugs.openjdk.org/browse/JDK-6832353
https://community.spiceworks.com/t/pam-keeps-setting-the-krb5ccname-env-variable/940232
https://linux.die.net/man/5/pam_krb5

分析直接原因

KRB5CCNAME 这个环境变量被改了，与实际的KRB5CCNAME不一致。

查找根本原因

启动脚本是通过 su - hue -c "springboot-app.jar start" 这种方式启动的

以前踩过一个坑 su - hue 执行的shell环境带hue环境变量，su hue 不带hue环境变量

最终原因

启动的脚本有错

#!/bin/bash

CURRENT_USER=$(whoami)
COMMAND='HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop /apps/springboot-app.jar start'
if [ $CURRENT_USER=='hue' ]; then
  echo "1. executing as $CURRENT_USER"
  bash -c "$COMMAND"
elif [ $CURRENT_USER=='root' ]; then
  echo "2. executing as hue... from $CURRENT_USER"
  su - hue -c "$COMMAND"
else
  echo "permission denied."
fi

结果发现是 if [ ] 表达式有错，打印的是

1. executing as hue

实际上执行的是 bash -c "$COMMAND" 而不是 su - hue -c "$COMMAND"

改正后成功打印

2. executing as hue... from root

结论

曾经怀疑过 su 是不是不支持 KERBEROS 相关的认证，结果证明是没问题的

只要认准 KRB5CCNAME 变量设置正确，就不会有问题。

最坑的是 Linux shell 语法，从 chatGPT 上复制尤其需要注意