首页 > 其他分享 >SeaTunnel Web 在 Debian 环境的安装

SeaTunnel Web 在 Debian 环境的安装

时间：2024-04-02 17:13:01浏览次数：12  
标签：bin Web SeaTunnel 1.0 web seatunnel datasource apache Debian

SeaTunnel Web 在 Debian 环境的安装

目录

环境说明

  • SeaTunnel 2.3.3
  • SeaTunnel Web 1.0.0
  • Debian 11.3

注: SeaTunnel Web 1.0.0 暂未适配 SeaTunnel 2.3.4 。

下载软件

下载SeaTunnel二进制文件
下载地址：https://seatunnel.apache.org/download/

  • apache-seatunnel-2.3.3-bin.tar.gz
  • apache-seatunnel-web-1.0.0-bin.tar.gz
    解压文件：
tar -xvf apache-seatunnel-2.3.3-bin.tar.gz
tar -xvf apache-seatunnel-web-1.0.0-bin.tar.gz

得到：

apache-seatunnel-2.3.3
apache-seatunnel-web-1.0.0-bin

安装目录

mv apache-seatunnel-2.3.3 /home/st/opt/apache-seatunnel-2.3.3
mv apache-seatunnel-web-1.0.0-bin /home/st/opt/apache-seatunnel-web-1.0.0-bin

下载连接器

进入 seatunnel 目录

cd apache-seatunnel-2.3.3

连接器配置列表：
文件路径： apache-seatunnel-2.3.3/config/plugin_config

建议初始下载连接器：

--connectors-v2--
connector-fake
connector-console
connector-cdc-mysql
--end--

下载连接器插件：

bash bin/install-plugin.sh 
或：
./bin/install-plugin.sh

注意： 不可以使用： sh bin/install-plugin.sh 以防解释器是 dash 而导致出错。
下载位置：
/opt/apache-seatunnel-2.3.3/connectors/seatunnel

注1： 经测试，SeaTunnel 2.3.4 与 SeaTunnel 2.3.3 下载连接器路径不同

2.3.3 : connectors/seatunnel
2.3.4 : connectors/

注 2：seatunnel web 1.0.0 将从 ${SEATUNNEL_HOME}/connectors/seatunnel 加载连接器插件.如果没有下载对应插件将导致配置任务时无法选择配置好的数据源。

注 3：install-plugin.sh

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#This script is used to download the connector plug-ins required during the running process. 
#All are downloaded by default. You can also choose what you need. 
#You only need to configure the plug-in name in config/plugin_config.

# get seatunnel home
SEATUNNEL_HOME=$(cd $(dirname $0);cd ../;pwd)

# connector default version is 2.3.3, you can also choose a custom version. eg: 2.1.2:  sh install-plugin.sh 2.1.2
version=2.3.3

if [ -n "$1" ]; then
    version="$1"
fi

echo "Install hadoop shade jar, usage version is ${version}"

${SEATUNNEL_HOME}/mvnw dependency:get -DgroupId=org.apache.seatunnel -Dclassifier=optional -DartifactId=seatunnel-hadoop3-3.1.4-uber -Dversion=${version} -Ddest=${SEATUNNEL_HOME}/lib

echo "Install SeaTunnel connectors plugins, usage version is ${version}"

# create the connectors directory
if [ ! -d ${SEATUNNEL_HOME}/connectors ];
  then
      mkdir ${SEATUNNEL_HOME}/connectors
      echo "create connectors directory"
fi

# create the seatunnel connectors directory (for v2)
if [ ! -d ${SEATUNNEL_HOME}/connectors/seatunnel ];
  then
      mkdir ${SEATUNNEL_HOME}/connectors/seatunnel
      echo "create seatunnel connectors directory"
fi  

while read line; do
    if  [ ${line:0:1} != "-" ] && [ ${line:0:1} != "#" ]
      	then
      		echo "install connector : " $line
      		${SEATUNNEL_HOME}/mvnw dependency:get -DgroupId=org.apache.seatunnel -DartifactId=${line} -Dversion=${version} -Ddest=${SEATUNNEL_HOME}/connectors/seatunnel
    fi

done < ${SEATUNNEL_HOME}/config/plugin_config

测试SeaTunnel任务

运行示例任务：

cd apache-seatunnel-2.3.3
./bin/seatunnel.sh --config ./config/v2.batch.config.template -e local

注： 如果你使用的是 SeaTunnel 2.3.4 ，因为seatunnel源码的问题，在Linux环境执行需要批量替换代码换行符为Unix格式

find apache-seatunnel-2.3.3 -type f -print0 | xargs -0 dos2unix --

下载数据库驱动

下载MySQL驱动并添加到lib目录
如：Mysql 驱动

cp mysql-connector-j-8.2.0.jar apache-seatunnel-2.3.3/lib/

注：从 Web 发起的数据库同步批任务将使用jdbc 方式连接。Seatunnel 如果没有对应驱动，任务执行将会连接失败。

启动客户端模式

mkdir logs
./bin/seatunnel-cluster.sh -d

配置web元数据库

cd apache-seatunnel-web-1.0.0-bin
cat script/seatunnel_server_env.sh

配置数据库连接信息。
注：此账号需要有创建数据库权限

export HOSTNAME="127.0.0.1"
export PORT="3306"
export USERNAME="root"
export PASSWORD="123456"

安装MySQL客户端

sudo apt install mycli

注：debian11 下没有 mysql-client

测试数据库连接：

mycli -h 127.0.0.1 -P 3306 -u root

修改初始化脚本：
script/init_sql.sh

workDir=`dirname $0`
workDir=`cd ${workDir};pwd`

source ${workDir}/seatunnel_server_env.sh

usage="Usage: seatunnel_server_env.sh must contain hostname/port/username/password."

if [[ ! -n "${HOSTNAME}" ]]  || [[ ! -n "${PORT}" ]] || [[ ! -n "${USERNAME}" ]] || [[ ! -n "${PASSWORD}" ]]; then
    echo $usage
    exit 1
fi

mycli -h${HOSTNAME} -P${PORT} -u${USERNAME} -p${PASSWORD} < ${workDir}/seatunnel_server_mysql.sql

设置数据库连接环境变量：

export HOSTNAME="localhost"
export PORT="3306"
export USERNAME="root"
export PASSWORD="123456"

初始化数据库:

sh script/init_sql.sh

配置Web应用

配置文件路径：conf/application.yml

cat conf/application.yml

修改内容：

datasource:
    # driver-class-name: com.mysql.jdbc.Driver
    driver-class-name: com.mysql.cj.jdbc.Driver
    url: jdbc:mysql://localhost:3306/seatunnel?useSSL=false&useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&allowPublicKeyRetrieval=true
    username: root
    password: 123456

注意：其中MySQL驱动类需要和实际的驱动类修改一致。
Mysql 8.0 及以上版本需使用 : driver-class-name: com.mysql.cj.jdbc.Driver
Mysql 5.0 等较低版本需使用：driver-class-name: com.mysql.jdbc.Driver

注：此步骤配置不正确，将导致无法在页面进行用户登录。

添加MySQL驱动

下载MySQL驱动并添加到libs目录

cp mysql-connector-j-8.2.0.jar apache-seatunnel-web-1.0.0-bin/libs/

配置 JAVA_HOME

echo "export JAVA_HOME=/path/to/java" >> ~/.profile
echo "export PATH=$JAVA_HOME/bin:$PATH" >> ~/.profile

配置 SEATUNNEL_HOME

echo "export SEATUNNEL_HOME=/home/st/opt/apache-seatunnel-2.3.3" >> ~/.profile

下载连接器到Web

下载或编写 download_datasource.sh
下载地址： https://seatunnel.apache.org/assets/files/download_datasource-4b79e6fafe80459590a6a0fc2865e5ac.sh

download_datasource.sh

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#This script is used to download the connector plug-ins required during the running process.
#All are downloaded by default. You can also choose what you need.
#You only need to configure the plug-in name in config/plugin_config.

# get seatunnel web home
SEATUNNEL_WEB_HOME=$(cd $(dirname $0);pwd)

DATASOURCE_LIB_DIR=${SEATUNNEL_WEB_HOME}/../libs

# the datasource default version is 1.0.0, you can also choose a custom version. eg: 1.1.2:  sh install-datasource.sh 2.1.2
version=1.0.0

if [ -n "$1" ]; then
    version="$1"
fi

# If you do not want to download a certain data source, you can delete the element below
datasource_list=(
  "datasource-plugins-api"
  "datasource-elasticsearch"
  "datasource-hive"
  "datasource-jdbc-clickhouse"
  "datasource-jdbc-hive"
  "datasource-jdbc-mysql"
  "datasource-jdbc-oracle"
  "datasource-jdbc-postgresql"
  "datasource-jdbc-redshift"
  "datasource-jdbc-sqlserver"
  "datasource-jdbc-starrocks"
  "datasource-jdbc-tidb"
  "datasource-kafka"
  "datasource-mysql-cdc"
  "datasource-s3"
  "datasource-sqlserver-cdc"
  "datasource-starrocks"
)

echo "Downloading SeaTunnel Web Datasource lib, usage version is ${version}"

for datasource in "${datasource_list[@]}"
do
    echo "Downloading datasource: ${datasource}"
  wget  https://repo.maven.apache.org/maven2/org/apache/seatunnel/${datasource}/${version}/${datasource}-${version}.jar -P $DATASOURCE_LIB_DIR
done

文件放置路径：

cp download_datasource.sh  apache-seatunnel-web-1.0.0-bin/bin/

下载连接器：

sh bin/download_datasource.sh

下载路径：

apache-seatunnel-web-1.0.0-bin/libs/

注：此步骤不执行将导致SeaTunnel Web 无法在页面配置添加数据源

复制连接器映射配置文件

映射文件在seatunnel安装目录下

cp apache-seatunnel-2.3.3/connectors/plugin-mapping.properties apache-seatunnel-web-1.0.0-bin/conf/

注：此文件用于配置 seatunnel 连接器与任务执行配置时 source 或 sink 的对应关系

启动服务

sh bin/seatunnel-backend-daemon.sh start

页面测试

页面访问：

localhost:8801/

默认账号：

user: admin
password: admin

常见问题

datasource-hive-1.0.0.jar 找不到类：

***************************
APPLICATION FAILED TO START
***************************

Description:

An attempt was made to call a method that does not exist. The attempt was made from the following location:

    org.springframework.boot.web.embedded.jetty.JettyServletWebServerFactory.configureSession(JettyServletWebServerFactory.java:267)

The following method did not exist:

    'void org.eclipse.jetty.server.session.SessionHandler.setMaxInactiveInterval(int)'

The calling method's class, org.springframework.boot.web.embedded.jetty.JettyServletWebServerFactory, was loaded from the following location:

    jar:file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/spring-boot-2.6.8.jar!/org/springframework/boot/web/embedded/jetty/JettyServletWebServerFactory.class

The called method's class, org.eclipse.jetty.server.session.SessionHandler, is available from the following locations:

    jar:file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar!/org/eclipse/jetty/server/session/SessionHandler.class
    jar:file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/jetty-server-9.4.46.v20220331.jar!/org/eclipse/jetty/server/session/SessionHandler.class

The called method's class hierarchy was loaded from the following locations:

    org.eclipse.jetty.server.session.SessionHandler: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
    org.eclipse.jetty.server.handler.ScopedHandler: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
    org.eclipse.jetty.server.handler.HandlerWrapper: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
    org.eclipse.jetty.server.handler.AbstractHandlerContainer: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
    org.eclipse.jetty.server.handler.AbstractHandler: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar
    org.eclipse.jetty.util.component.ContainerLifeCycle: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/jetty-util-9.4.46.v20220331.jar
    org.eclipse.jetty.util.component.AbstractLifeCycle: file:/home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/jetty-util-9.4.46.v20220331.jar

原因：未知
解决方法：删除 datasource-hive-1.0.0.jar

rm /home/st/opt/apache-seatunnel-web-1.0.0-bin/libs/datasource-hive-1.0.0.jar

标签：bin,Web,SeaTunnel,1.0,web,seatunnel,datasource,apache,Debian
From： https://www.cnblogs.com/nookvoice/p/18110597

相关文章

  • webscoket 中的 mask 细节
    掩码用于给客户端到服务端的帧数据加密（异或的方式，非常简单），对此RFC6455中给了一些细节如下：Themaskingkeyiscontainedcompletelywithintheframe,asdefinedinSection5.2asframe-masking-key.Itisusedtomaskthe"Payloaddata"definedinthesamesection......
  • 海豚调度任务类型Apache SeaTunnel部署指南
    ApacheDolphinScheduler已支持ApacheSeaTunnel任务类型，本文介绍了SeaTunnel任务类型如何创建，任务参数，以及任务样例。一、ApacheSeaTunnelSeaTunnel任务类型，用于创建并执行SeaTunnel类型任务。worker执行该任务的时候，会通过start-seatunnel-spark.sh、start-seatunnel......
  • 【javaWeb & 功能介绍第一篇】阿里云OSS文件上传
    文件上传文件上传存储文件本地存储云服务阿里云文件上传文件上传是将本地的图片，视频，音频等文件上传到服务器，供其他用户浏览或下载的过程文件上传在项目中应用十分广泛，我们经常发微博，发微信都用到了文件上传的功能在前端的开发之中，如果需要文件上传功能，则必须在......
  • SeaTunnel DB2 Source Connector 使用文档(含详细操作步骤)
    DB2是IBM的一款关系型数据库管理系统，JDBCDB2SourceConnector是一个用于通过JDBC读取外部数据源数据的连接器。ApacheSeaTunnel如何支持JDBCDB2SourceConnector？请参考本文档。支持引擎SparkFlinkSeaTunnelZeta引擎主要特性批处理（batch）精确一次（exactly-once）列投......
  • Yarn介绍及快速安装Debian/Ubuntu Linux
    Yarn简介：Yarn是一个快速、可靠、安全的JavaScript软件包管理工具。它是由Facebook开发的，旨在改进npm（NodePackageManager）的性能和可靠性。Yarn提供了更快的下载速度、离线支持、版本锁定等功能，使JavaScript项目的依赖管理更加高效。在Debian/UbuntuLinux上快速安装Yarn：要在De......
  • .net core webapi统一修改日期时间的返回格式
    在写WebApi时，我们的日期时间往往是DateTime类型，如果直接返回，前端拿到的结果有可能会是这种格式：下面是如何将返回的DateTime字段统一改成自定义的格式首先创建一个自定义的JSON序列化器，定义一个继承System.Text.Json.Serialization.JsonConverter的类，实现Read和Write两个抽象方......
  • 54.html+css+js网页设计实例/“企业”酒庄主题介绍/web前端期末大作业/
    一、前言  本实例以“企业”酒庄为主题设计，应用html+css+js、图片轮翻效果、留言板、搜索等，供大家参考。【关注作者|获取更多源码（2000+个Web案例源码）|优质文章】；您的支持是我创作的动力！【点赞收藏博文】，Web开发、课程设计、毕业设计有兴趣的联系我交流分享，3Q！二、网页文......
  • COMP S380F Web应用程序的设计与开发
    COMPS380F集团项目（2024）COMPS380FWeb应用程序的设计与开发集团项目（15%）您需要组成一个最多由4名成员组成的小组，每个成员都应分担类似的工作量。主题：您需要为在线书店实现web应用程序。基本特征（占项目的60%）：1.您的web应用程序应满足以下关于网页和功能的基本要求：a.使用讲座和实验室......
  • JavaWeb学习笔记——第十二天
    SpringBootWeb案例（三）登录功能LoginController：importcom.zgg1h.pojo.Emp;importcom.zgg1h.pojo.Result;importcom.zgg1h.service.EmpService;importlombok.extern.slf4j.Slf4j;importorg.springframework.beans.factory.annotation.Autowired;importorg.springfram......
  • 零基础10 天入门 Web3之第2天
    10天入门Web3之第2天Web3 是互联网的下一代，它将使人们拥有自己的数据并控制自己的在线体验。Web3基于区块链技术，该技术为安全、透明和可信的交易提供支持。我准备做一个10天的学习计划，可帮助大家入门Web3：一、这是第二天，首先免费下载一下两篇白皮书尽量一口气看完它：比......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99