首页 > 其他分享 >Flink 1.17教程:Hadoop yarn运行模式——单作业模式和应用模式

Flink 1.17教程:Hadoop yarn运行模式——单作业模式和应用模式

时间:2023-09-02 11:36:04浏览次数:45  
标签:1.17 Flink flink jar yarn 模式 atguigu org


YARN运行模式_单作业模式

单作业模式部署

(1)执行命令提交作业

Flink 1.17教程:Hadoop yarn运行模式——单作业模式和应用模式_hadoop

YARN运行模式_应用模式

应用模式同样非常简单,与单作业模式类似,直接执行

flink run-application

命令即可。如:

bin/flink run-application -t yarn-application -c com.atguigu.wc.WordCountStreamUnboundedDemo ./FlinkTutorial-1.17-1.0-SNAPSHOT.jar
[atguigu@node001 flink-1.17.0]$ bin/flink run-application -t yarn-application -c com.atguigu.wc.WordCountStreamUnboundedDemo ./FlinkTutorial-1.17-1.0-SNAPSHOT.jar 

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/flink/flink-1.17.0/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2023-06-19 14:31:05,693 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Found Yarn properties file under /tmp/.yarn-properties-atguigu.
2023-06-19 14:31:05,693 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Found Yarn properties file under /tmp/.yarn-properties-atguigu.
2023-06-19 14:31:06,142 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/opt/module/flink/flink-1.17.0/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2023-06-19 14:31:06,632 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at node002/192.168.10.102:8032
2023-06-19 14:31:07,195 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar

Flink 1.17教程:Hadoop yarn运行模式——单作业模式和应用模式_大数据_02

Flink 1.17教程:Hadoop yarn运行模式——单作业模式和应用模式_hadoop_03

jar存放在hdfs的情况:

[atguigu@node001 flink-1.17.0]$ bin/flink run-application -t yarn-application -Dyarn.provided.lib.dirs="hdfs://node001:8020/flink-dist" -c com.atguigu.wc.WordCountStreamUnboundedDemo hdfs://node001:8020/flink-jars/FlinkTutorial-1.17-1.0-SNAPSHOT.jar


标签:1.17,Flink,flink,jar,yarn,模式,atguigu,org
From: https://blog.51cto.com/zhangxueliang/7331340

相关文章

  • Flink 1.17教程:命令行提交作业jar
    命令行提交作业bin/flinkrun-mnode001:8081-ccom.atguigu.wc.WordCountStreamUnboundedDemo../jar/FlinkTutorial-1.17-1.0-SNAPSHOT.jar连接成功Lastlogin:FriJun1614:44:012023from192.168.10.1[atguigu@node001~]$cd/opt/module/flink/flink-1.17.0/[atgu......
  • Flink 1.17教程:集群搭建、运行模式(standalone/yarn/k8s)及历史服务器
    集群角色集群启动如果是部署在本地,本地访问,无需进行任何配置,直接启动即可。如果是部署在服务器,需要远程访问,则需要将flink.conf中的localhost修改为服务器IP地址或是0.0.0.0节点服务器hadoop102hadoop103hadoop104角色JobManagerTaskManagerTaskManagerTaskManager[atguigu@node001......
  • Flink 1.17教程:WebUI提交作业及打jar包maven插件配置
    打jar包maven插件配置<build><plugins><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-shade-plugin</artifactId><version>3.2.4</version>......
  • Flink 1.17教程:集群角色及集群启动
    集群角色集群启动如果是部署在本地,本地访问,无需进行任何配置,直接启动即可。如果是部署在服务器,需要远程访问,则需要将flink.conf中的localhost修改为服务器IP地址或是0.0.0.0节点服务器hadoop102hadoop103hadoop104角色JobManagerTaskManagerTaskManagerTaskManager[atguigu@node001......
  • Flink 1.17教程:wordcount maven工程java代码示例(批、流实现方式)
    批、流实现wordcount代码示例pom.xml<properties><flink.version>1.17.0</flink.version></properties><dependencies><dependency><groupId>org.apache.flink</groupId><......
  • Flink 1.17教程:DataStream实现Wordcount——读socket(无界流)
    pom.xml<properties><flink.version>1.17.0</flink.version></properties><dependencies><dependency><groupId>org.apache.flink</groupId><artifactId>fli......
  • Java代码:flink wordcount代码示例及解读
    WordCountWordCountPojo.java代码packagewordCount;importorg.apache.flink.api.common.functions.FlatMapFunction;importorg.apache.flink.api.common.functions.ReduceFunction;importorg.apache.flink.api.java.DataSet;importorg.apache.flink.api.java.Executio......
  • 230901 简单模式,避免贪婪
    短线操作中,存在各种各样的模式与上涨方式.比如德赛西威的机构缓慢上涨, 星期六,中贝,金科股份,直接快速上涨.美丽生态,金盛控股,人民网的震荡上涨方式.多种上涨方式,则存在多种的买入位置与,买点与卖点.你要想把这些所有的都覆盖,显然是不可能的.比如,掌握打板,掌握半路,......
  • 状态模式-22
    概述状态模式(StatePattern)又称状态对象(ObjectsforStates)。当一个对象的内部状态改变时其行为跟着改变。优点:提高可维护性。缺点:增加了类的数量,实现较复杂,不符合“开闭原则”。classContext{privateStatestate;publicsetState(States){state=s;}......
  • 设计模式:观察者模式/发布-订阅模式
    设计模式wiki中将设计模式分为四类,分别是:创建模式(creationalpatterns)结构模式(structuralpatterns)行为模式(behavioralpatterns)并发模式(concurrencypatterns)观察者/发布订阅模式属于其中的行为模式。实际情境公众号订阅说到发布订阅,拿微信公众号举例,就很好理解,有n个......