首页 > 系统相关 >【干货】解决CentOS 7.x上安装与部署SeaTunnel Web时遇到的问题

【干货】解决CentOS 7.x上安装与部署SeaTunnel Web时遇到的问题

时间：2024-01-30 16:01:30浏览次数：28  
标签：11 Web SeaTunnel 12 CentOS 19 seatunnel 2023 apache

1. 环境

采用的是虚拟机上安装CentOs7.x，安装java15环境，安装mysql8.0.28，这两步骤省略，之前的文章里面有讲，属于基础环境配置，也比较简单，说明我用的是一台CentOs7.x的虚拟机实例，都安装在一台上面，需要开放8081、3306和5801的防火墙，不然会网络访问不到。

2. SeaTunnel安装部署

2.1下载安装包

export version="2.3.3"
wget "https://archive.apache.org/dist/seatunnel/${version}/apache-seatunnel-${version}-bin.tar.gz"
tar -xzvf "apache-seatunnel-${version}-bin.tar.gz"

2.2 设置环境变量

vi /etc/profile.d/seatunnel.sh
# 新增如下变量
export SEATUNNEL_HOME=/root/apache-seatunnel-2.3.3 #这里设置的是seatunnel的解压目录即可
export PATH=$PATH:$SEATUNNEL_HOME/bin

然后执行source /etc/profile

2.3 安装连接器插件

进入/root/apache-seatunnel-2.3.3路径下执行安装连接器插件

file

sh bin/install-plugin.sh 2.3.3

可以修改/root/apache-seatunnel-2.3.3/connectors下的plugin-mapping.properties文件，根据需要加入连接器插件，把不需要的连接器插件名称删除留下需要的即可，然后执行上面的安装命令，由于我没有修改这个文件，默认安装的是全部的连接器插件，所以需要一点时间，下载速度取决于网速。

2.4 拷贝jar包到lib下

file

2.5 启动命令

进入/root/apache-seatunnel-2.3.3路径下执行启动命令：

sh bin/seatunnel-cluster.sh -d  -DJvmOption="-Xms1G -Xmx1G"
或者
nohup sh bin/seatunnel-cluster.sh 2>&1 &

jps查看seatunnel的进程如下：

file

查看logs下的日志均正常输出没有报错，说明启动成功，然后执行下面的官方demo没有报错，说明seatunnel启动是正常的。

2.6 执行官方client提交任务demo

进入/root/apache-seatunnel-2.3.3路径下执行启动命令：

$SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template

该命令来自于官网，执行结果如下：

[root@es1 apache-seatunnel-2.3.3]# $SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/root/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /root/apache-seatunnel-2.3.3/config/seatunnel.yaml
12月 19, 2023 12:11:20 上午 org.apache.seatunnel.engine.common.config.SeaTunnelConfig
信息: seatunnel.home is /root/apache-seatunnel-2.3.3
12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/root/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /root/apache-seatunnel-2.3.3/config/hazelcast.yaml
12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/root/apache-seatunnel-2.3.3/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /root/apache-seatunnel-2.3.3/config/hazelcast-client.yaml
2023-12-19 00:11:21,149 INFO  com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true
2023-12-19 00:11:21,233 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2023-12-19 00:11:21,234 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/root/apache-seatunnel-2.3.3/starter/seatunnel-starter.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2023-12-19 00:11:21,294 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel
2023-12-19 00:11:21,298 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to [localhost]:5801
2023-12-19 00:11:21,352 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2023-12-19 00:11:21,352 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Authenticated with server [localhost]:5801:772efc0a-4c18-4a4b-baa7-b82b9ae4a395, server version: 5.1, local address: /127.0.0.1:36095
2023-12-19 00:11:21,356 INFO  com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-12-19 00:11:21,384 INFO  com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel] [5.1] 

Members [1] {
    Member [localhost]:5801 - 772efc0a-4c18-4a4b-baa7-b82b9ae4a395
}

2023-12-19 00:11:21,421 INFO  com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
2023-12-19 00:11:21,706 INFO  org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[]
2023-12-19 00:11:21,733 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: /root/apache-seatunnel-2.3.3/config/v2.batch.config.template
2023-12-19 00:11:21,799 INFO  org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64]
2023-12-19 00:11:21,848 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
    "env" : {
        "execution.parallelism" : 2,
        "job.mode" : "BATCH",
        "checkpoint.interval" : 10000
    },
    "source" : [
        {
            "schema" : {
                "fields" : {
                    "name" : "string",
                    "age" : "int"
                }
            },
            "row.num" : 16,
            "parallelism" : 2,
            "result_table_name" : "fake",
            "plugin_name" : "FakeSource"
        }
    ],
    "sink" : [
        {
            "plugin_name" : "Console"
        }
    ]
}

2023-12-19 00:11:21,885 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:21,886 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:21,895 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /root/apache-seatunnel-2.3.3/connectors/seatunnel
2023-12-19 00:11:21,911 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar
2023-12-19 00:11:21,912 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Console at: file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar
2023-12-19 00:11:21,915 INFO  org.apache.seatunnel.engine.core.parse.ConfigParserUtil - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations.
2023-12-19 00:11:21,915 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:21,915 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:21,916 WARN  org.apache.seatunnel.engine.core.parse.ConfigParserUtil - This configuration is not recommended. A source/transform(FakeSource) is configured with 'result_table_name' option value of 'fake', but subsequent transform/sink(Console) is not configured with 'source_table_name' option.
2023-12-19 00:11:21,919 INFO  org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sources.
2023-12-19 00:11:21,919 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:21,953 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /root/apache-seatunnel-2.3.3/connectors/seatunnel
2023-12-19 00:11:21,970 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar
2023-12-19 00:11:21,974 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='FakeSource'} from classpath
2023-12-19 00:11:22,003 INFO  org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all transforms.
2023-12-19 00:11:22,003 INFO  org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sinks.
2023-12-19 00:11:22,004 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:22,011 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-19 00:11:22,090 INFO  org.apache.seatunnel.engine.client.job.ClientJobProxy - Start submit job, job id: 789162834679300097, with plugin jar [file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar, file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar]
2023-12-19 00:11:22,893 INFO  org.apache.seatunnel.engine.client.job.ClientJobProxy - Submit job finished, job id: 789162834679300097, job name: SeaTunnel
2023-12-19 00:11:22,956 WARN  org.apache.seatunnel.engine.client.job.JobMetricsRunner - Failed to get job metrics summary, it maybe first-run
2023-12-19 00:11:24,370 INFO  org.apache.seatunnel.engine.client.job.ClientJobProxy - Job (789162834679300097) end with state FINISHED
2023-12-19 00:11:24,416 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - 
***********************************************
           Job Statistic Information
***********************************************
Start Time                : 2023-12-19 00:11:21
End Time                  : 2023-12-19 00:11:24
Total Time(s)             :                   2
Total Read Count          :                  32
Total Write Count         :                  32
Total Failed Count        :                   0
***********************************************

2023-12-19 00:11:24,416 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2023-12-19 00:11:24,422 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [localhost]:5801:772efc0a-4c18-4a4b-baa7-b82b9ae4a395, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:36095->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2023-12-19 00:11:24.411, lastWriteTime=2023-12-19 00:11:24.371, closedTime=2023-12-19 00:11:24.420, connected server version=5.1}
2023-12-19 00:11:24,422 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2023-12-19 00:11:24,431 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2023-12-19 00:11:24,433 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
2023-12-19 00:11:24,433 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed metrics executor service ......
2023-12-19 00:11:24,438 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal

3. SeaTunnel-Web安装部署

3.1 下载安装包

安装包下在地址如下：

https://seatunnel.apache.org/download

解压：

tar -zxvf apache-seatunnel-web-bin-${project.version}.tar.gz

解压目录如下：

file

3.2 初始化数据库脚本或修改配置application.yml中的数据库连接信息

3.2.1 不使用它的初始化脚本自动创建

事先，手动执行好脚本，然后在修改application.yml文件中的数据库连接信息

3.2.2 使用脚本初始化数据库

首先，设置变量如下：

export HOSTNAME="localhost"
export PORT="3306"
export USERNAME="root"
export PASSWORD="123456"

然后执行：

sh apache-seatunnel-web-bin-2.3.3/script/init_sql.sh

如果环境变量有冲突需要改下名字和init_sql.sh中的环境变量的名字，可以加一个前缀可以加上前缀`STWEB_ ，然后把init_sql.sh中上面的那几个环境变量也加上这个前缀即可，然后执行初始化命令

3.3 修改端口与数据源

vim conf/application.yml修改端口号和数据源信息

file

3.4 两个文件拷贝配置

需要将apache-seatunnel-2.3.3/config/hazelcast-client和apache-seatunnel-2.3.3/connectors/plugin-mapping.properties文件拷贝到apache-seatunnel-web-bin-2.3.3的conf下。

3.5 拷贝jar到lib下

file

3.6 启动

sh bin/seatunnel-backend-daemon.sh start

jps查看java进程如下：

file

这里有个很坑的地方就是，执行命令如果是进入bin路径执行，访问首页的时候就会有问题，访问首页就会报404的错

sh seatunnel-backend-daemon.sh start

访问首页报错如下：

file

3.7 访问首页

访问ip:8081/ui，此端口为conf/application.yml中配置的端口

http://192.168.1.4:8081/

file

登录不上有可能是没有启动mysql数据库

service mysqld start   #使用service启动mysql服务
service mysqld status  #查看MYSQL运行状态
service mysqld stop    #使用service停止mysql服务
service mysqld restart #使用service重启mysql服务
systemctl enable mysqld.service #设置MySQL服务开机自动启动
systemctl is-enabled mysqld.service #确认MySQL服务已经被设置为自动启动

file

3.8 执行myslq-jdbc-->mysl-jdbc的单表数据同步

执行是成功的，但是在我的CentOs7.x的虚拟机上我是没有安装hadoop3.1.3的环境，执行看日志是没有报错的，官方说是这个hadoop的环境不是必须的也是有道理的，但是之前的文章在本地调试的时候没有hadoop环境会报错，所以建议需要本地编译构建的小伙伴还是安装一个hadoop，以免执行任务报错。

4. 总结

本次分享就到此结束了，踩了一些坑，还是请教了官方的大佬才知道的是哪里有问题，希望我的分享对你有所帮助，请一键三连，么么么哒！

本文由 白鲸开源 提供发布支持！

标签：11,Web,SeaTunnel,12,CentOS,19,seatunnel,2023,apache
From： https://blog.51cto.com/u_15459354/9488960

相关文章

  • 新增一个web服务
    1. cd/wenzd/ 2. cpdigg_update_publish.shmpapi_update_publish.sh复制（新建）命令到当前目录。 3.修改命令里面的内容 4.到服务目录：cd/etc/systemd/system复制（新建）文件：cpcentral.api.servicemp.api.service编辑服务文件：vi mp.api.service5. 6. ......
  • 从零搭建Go语言Web服务器
    从零搭建Go语言Web服务器原创 Go先锋 Go先锋 2024-01-3011:19 发表于广东 听全文Go先锋读完需要9分钟速读仅需3分钟  一、Go语言的优势1.执行效率高Go语言以其出色的执行效率而闻名，这得益于其静态类型和编译型的特性。通过直接编译成机器码，Go程序......
  • CentOS启动nginx出现nginx: [emerg] open() "/var/run/nginx/nginx.pid" failed (2: N
    问题：1.进入sbin：cd/usr/local/nginx/sbin/2.启动nginx：./nginx3.出现问题：nginx:[emerg]open()"/var/run/nginx/nginx.pid"failed(2:Nosuchfileordirectory) 原因：没有nginx文件夹，且其下没有nginx.pid文件。 解决办法：创建文件1.进入run下：cd/var/run2.创建nginx......
  • 最简单的centos搭建frp内网穿透
    最简单的centos搭建frp内网穿透 https://www.cnblogs.com/phpwyl/p/16466531.html首先服务端搭建1.进入软件安装目录cd/usr/local/src2.下载frp版本可以自己选择，如果下载慢，可以直接通过浏览器或挂代理下载wgethttps://github.com/fatedier/frp/releases/download/v......
  • WebAssembly入门笔记[4]：利用Global传递全局变量
    利用WebAssembly的导入导出功能可以灵活地实现宿主JavaScript程序与加载的单个wasm模块之间的交互，那么如何在宿主程序与多个wasm之间传递和共享数据呢？这就需要使用到Global这个重要的对象了。一、数值类型全局变量二、将JavaScript函数设置为全局变量三、利用全局变量处理字符......
  • websocket客户端类，工具类中调用到service层去
    maven<dependency><groupId>org.java-websocket</groupId><artifactId>Java-WebSocket</artifactId><version>1.3.5</version></dependency>代码@Slf4j@ComponentpublicclassWESWebSocketClientexten......
  • [CISCN2019 华北赛区 Day2 Web1]Hack World
    [CISCN2019华北赛区Day2Web1]HackWorld页面中给了提示，数据表名和列名都为flag输入1尝试输入2尝试输入3提示获取结果出错输入1'尝试找注入点，根据返回结果判断可能是字符型注入使用永真语句尝试发现注入被过滤通过爆破发现部分关键字并没有被过滤这里通过师傅们（h......
  • 黄金web架构之LAMP
    什么是LAMPLinux操作系统Apache/Nginxweb服务器Mysql/MariadbPerl/Php/Python部署LAMP实战环境准备关于基础软件的配置大前提是，你的yum源配置好了吗？建议用最新的阿里云yum源[root@client-242~]#yuminstallwget-y再去配置阿里云的yum源https://developer.......
  • WebSocket和RESTful API区别
    1.WebSocket和RESTfulAPI区别WebSocket和RESTfulAPI是用于在客户端和服务器之间进行通信的不同协议。RESTfulAPI（RepresentationalStateTransfer）是一种使用HTTP协议进行通信的架构风格。它基于客户端-服务器模型，通过使用不同的HTTP动词（GET、POST、PUT、DELETE等）对资源进行......
  • CTF常用工具 | MISC & CRYPTO & WEB
    记录一些做题过程用到的工具，不定时更新MISC010Editor：查看及处理十六进制文件StegSolve：查看及处理隐写图片Stegsolve（隐写分析工具）-宇2468-博客园WaterMarkH：添加及提取图片盲水印，提取不需要原图BlindWaterMark-master：添加及提取图片盲水印，提取时需要原图【Python2Python3......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99