首页 > 其他分享 >Flink1.15仅支持ZooKeeper3.5/3.6

Flink1.15仅支持ZooKeeper3.5/3.6

时间:2022-12-08 19:01:00浏览次数:110  
标签:1.15 java ZooKeeper3.5 flink jar 3.6 apache Flink1.15 org

这是一个验证贴,因为社区文档是错误的。

先说结论

Flink1.15仅支持ZooKeeper3.5/3.6,不再支持3.4。FLINK-25146 Drop support for Zookeeper 3.4 

https://nightlies.apache.org/flink/flink-docs-release-1.16/release-notes/flink-1.15

社区文档

其中的ZooKeeper Versions描述默认使用3.4,可以自行调整为3.5。从文档上来看自1.11版本就添加了这段描述,到目前最新稳定版本(1.16.0)依然是这么描述的。但是实际我们在flink-1.15.0/lib目录中只看到了flink-shaded-zookeeper-3.5.9.jar,在opt目录中看到了flink-shaded-zookeeper-3.6.3.jar,即默认使用3.5版本,并在opt目录中放置了3.6版本。

其实在1.15的 ReleaseNote 中明确写了,不再支持3.4,只是还是有点不死心,做了如下的验证。

Drop support for Zookeeper 3.4 #

Support for using Zookeeper 3.4 for HA has been dropped. Users relying on Zookeeper need to upgrade to 3.5/3.6. By default Flink now uses a Zookeeper 3.5 client.

 

 

验证一

那么我们是否可以将3.4的版本(flink-1.13.3/lib/flink-shaded-zookeeper-3.4.14.jar)放置到flink-1.15.3/lib目录中,在1.15版本中使用3.4版本呢?

不能

如果我们这么做,将会遇到 java.lang.ClassNotFoundException: org.apache.flink.shaded.curator5.org.apache.curator.framework.api.ACLProvider ,完整堆栈如下:

 1 2022-12-08 18:18:05,693 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Stopping Akka RPC service.
 2 2022-12-08 18:18:05,734 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator        [] - Shutting down remote daemon.
 3 2022-12-08 18:18:05,736 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator        [] - Remote daemon shut down; proceeding with flushing remote transports.
 4 2022-12-08 18:18:05,758 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator        [] - Remoting shut down.
 5 2022-12-08 18:18:05,782 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Stopped Akka RPC service.
 6 2022-12-08 18:18:05,782 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Could not start cluster entrypoint YarnApplicationClusterEntryPoint.
 7 org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint YarnApplicationClusterEntryPoint.
 8     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:250) ~[flink-dist-1.15.3.jar:1.15.3]
 9     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:711) [flink-dist-1.15.3.jar:1.15.3]
10     at org.apache.flink.yarn.entrypoint.YarnApplicationClusterEntryPoint.main(YarnApplicationClusterEntryPoint.java:112) [flink-dist-1.15.3.jar:1.15.3]
11 Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/curator5/org/apache/curator/framework/api/ACLProvider
12     at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createZooKeeperHaServices(HighAvailabilityServicesUtils.java:90) ~[flink-dist-1.15.3.jar:1.15.3]
13     at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:140) ~[flink-dist-1.15.3.jar:1.15.3]
14     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:427) ~[flink-dist-1.15.3.jar:1.15.3]
15     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:376) ~[flink-dist-1.15.3.jar:1.15.3]
16     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:277) ~[flink-dist-1.15.3.jar:1.15.3]
17     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:227) ~[flink-dist-1.15.3.jar:1.15.3]
18     at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_202]
19     at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_202]
20     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) ~[hadoop-common-3.0.0-cdh6.3.2.jar:?]
21     at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist-1.15.3.jar:1.15.3]
22     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:224) ~[flink-dist-1.15.3.jar:1.15.3]
23     ... 2 more
24 Caused by: java.lang.ClassNotFoundException: org.apache.flink.shaded.curator5.org.apache.curator.framework.api.ACLProvider
25     at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_202]
26     at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_202]
27     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_202]
28     at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_202]
29     at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createZooKeeperHaServices(HighAvailabilityServicesUtils.java:90) ~[flink-dist-1.15.3.jar:1.15.3]
30     at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:140) ~[flink-dist-1.15.3.jar:1.15.3]
31     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:427) ~[flink-dist-1.15.3.jar:1.15.3]
32     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:376) ~[flink-dist-1.15.3.jar:1.15.3]
33     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:277) ~[flink-dist-1.15.3.jar:1.15.3]
34     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:227) ~[flink-dist-1.15.3.jar:1.15.3]
35     at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_202]
36     at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_202]
37     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) ~[hadoop-common-3.0.0-cdh6.3.2.jar:?]
38     at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist-1.15.3.jar:1.15.3]
39     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:224) ~[flink-dist-1.15.3.jar:1.15.3]
40     ... 2 more

 

 

验证二

如果使用3.5的客户端访问3.4版本服务端,将会提示 org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for 异常,完整堆栈如下:

 1 2022-12-08 15:10:14,935 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Fatal error occurred in the cluster entrypoint.
 2 org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /flink/jm_ha_metadata/application_1666841155139_0173
 3     at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:106) ~[flink-shaded-zookeeper-3-3.5.9-15.0.jar:3.5.9-15.0]
 4     at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[flink-shaded-zookeeper-3-3.5.9-15.0.jar:3.5.9-15.0]
 5     at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538) ~[flink-shaded-zookeeper-3-3.5.9-15.0.jar:3.5.9-15.0]
 6     at org.apache.flink.shaded.curator5.org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:351) ~[flink-shaded-zookeeper-3-3.5.9-15.0.jar:3.5.9-15.0]
 7     at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90) ~[flink-shaded-zookeeper-3-3.5.9-15.0.jar:3.5.9-15.0]
 8     at org.apache.flink.shaded.curator5.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93) ~[flink-shaded-zookeeper-3-3.5.9-15.0.jar:3.5.9-15.0]
 9     at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83) ~[flink-shaded-zookeeper-3-3.5.9-15.0.jar:3.5.9-15.0]
10     at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.NamespaceImpl.newNamespaceAwareEnsurePath(NamespaceImpl.java:109) ~[flink-shaded-zookeeper-3-3.5.9-15.0.jar:3.5.9-15.0]
11     at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.newNamespaceAwareEnsurePath(CuratorFrameworkImpl.java:618) ~[flink-shaded-zookeeper-3-3.5.9-15.0.jar:3.5.9-15.0]
12     at org.apache.flink.runtime.util.ZooKeeperUtils.useNamespaceAndEnsurePath(ZooKeeperUtils.java:729) ~[flink-runtime-1.15.2.jar:1.15.2]
13     at org.apache.flink.runtime.highavailability.zookeeper.ZooKeeperMultipleComponentLeaderElectionHaServices.<init>(ZooKeeperMultipleComponentLeaderElectionHaServices.java:85) ~[flink-runtime-1.15.2.jar:1.15.2]
14     at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createZooKeeperHaServices(HighAvailabilityServicesUtils.java:96) ~[flink-runtime-1.15.2.jar:1.15.2]
15     at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:140) ~[flink-runtime-1.15.2.jar:1.15.2]
16     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:427) ~[flink-runtime-1.15.2.jar:1.15.2]
17     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:376) ~[flink-runtime-1.15.2.jar:1.15.2]
18     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:277) ~[flink-runtime-1.15.2.jar:1.15.2]
19     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:227) ~[flink-runtime-1.15.2.jar:1.15.2]
20     at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_202]
21     at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_202]
22     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) [hadoop-common-3.0.0-cdh6.3.2.jar:?]
23     at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) [flink-runtime-1.15.2.jar:1.15.2]
24     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:224) [flink-runtime-1.15.2.jar:1.15.2]
25     at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:711) [flink-runtime-1.15.2.jar:1.15.2]
26     at org.apache.flink.yarn.entrypoint.YarnApplicationClusterEntryPoint.main(YarnApplicationClusterEntryPoint.java:112) [flink-dist-1.15.3.jar:1.15.3]
27 2022-12-08 15:10:14,943 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting YarnApplicationClusterEntryPoint down with application status UNKNOWN. Diagnostics Cluster entrypoint has been closed externally..

 

 

参考资料:

  1. FLINK-25146Drop support for Zookeeper 3.4
  2. ZooKeeper Version 3.4.x Compatibility

  3. https://nightlies.apache.org/flink/flink-docs-release-1.16/release-notes/flink-1.15/

标签:1.15,java,ZooKeeper3.5,flink,jar,3.6,apache,Flink1.15,org
From: https://www.cnblogs.com/aaronking/p/16966998.html

相关文章

  • [Zookeeper-3.6.2源码解析系列]-13-Zookeeper使用到的Reactor网络模型原理分析
    目录​​13-启服务端网络监听连接NIOServerCnxnFactory​​​​13.1简介​​​​13.2主从Reactor网络IO模型main-subreactor​​​​13.3NIOServerCnxnFactory的初始化......
  • 3.6 Docker最新入门教程-Docker入门-使用绑定挂载
    3.6使用绑定挂载在上一章中,我们讨论并使用命名卷来持久化数据库中的数据。如果我们只想存储数据,命名卷就很棒,因为我们不必担心数据存储在哪里。使用绑定挂载,我们可以控......
  • FileNotFoundError: [Errno 2] No such file or directory: ‘sample_weight.pkl‘《
    报错的意思是没有找到“sample_weight.pkl”这个文件的地址或者路径解决方法:将本书的源代码文件夹改名为Deeplearningfromscratchcode并存放至jupyternotebook存放文......
  • python 3.6下 安装mysqlclient
    倒腾了大半天,终于把mysqlclient安装成功,赶紧把步骤进行梳理并记录1、python安装好以后,首先要看一下自己的python是32还是64位的,这关系到你下载的mysqlclient。查看方法:......
  • 【tensorflow2.6】图片数据建模流程:猫狗分类,83.6%识别率
    目标:识别猫和狗一、猫狗数据集数据集下载:公众号,回复:猫狗数据集训练数据集(每一张图片都有dog和cat标签):测试集(图片没有标签):二、训练环境kaggletenslrflow2.6三、数据处理impo......
  • 从零开始游戏开发——3.6 材质与Mesh
    需要显示一个模型,虽然可以通过在代码指定顶点数据、显示的纹理、使用的Shader等内容,但游戏开中这些数据通常都是由美术人员提供的,这就涉及到了材质和Mesh的概念。材质......
  • 老版本Python3.6 安装踩坑
    因业务需要安装老版本的python3.6.7,当前系统默认的python2.7brew安装的3.7 brew回退安装3.6.*版本的方式,网络上有很多。比较靠谱的可以用官网方式安装,见 ​​ht......
  • win10+VS2017+OSG3.6.4+OSGEarth2.10.2
    谨记:配完环境变量后最好重启一下电脑再测试,否则可能发生运行找不到dll报错的问题。 cmake3.15.6:https://cmake.org/files/v3.15/cmake-3.15.6-win64-x64.msi【until:2......
  • 二进制方式部署K8S-v1.23.6(中)
    5、部署k8s5-1、下载安装包#master-101执行:#下载Kubernetes软件包并解压安装(建议使用某雷下载),此处以v1.23.6为例:#下载地址:https://dl.k8s.io/v1.23.6/kubernetes-server-l......
  • Centos8 升级python3.6版本到python3.9
    下载Python3.9.51:wgethttps://www.python.org/ftp/python/3.9.5/Python-3.9.5.tgz2:tarzxvfPython-3.9.5.tgz安装必须的包3:dnf-yinstallgcczlib*libffi-dev......