seatunnle安装部署

环境准备

安装jdk8或jdk11

在线安装

下载seatunnle

export version="2.3.3"
wget "https://archive.apache.org/dist/seatunnel/${version}/apache-seatunnel-${version}-bin.tar.gz"
tar -xzvf "apache-seatunnel-${version}-bin.tar.gz"　

安装连接器

sh bin/install-plugin.sh 2.3.3

配置验证任务

env {
  execution.parallelism = 1
  job.mode = "BATCH"
}

source {
  FakeSource {
    result_table_name = "fake"
    row.num = 16
    schema = {
      fields {
        name = "string"
        age = "int"
      }
    }
  }
}

transform {
  FieldMapper {
    source_table_name = "fake"
    result_table_name = "fake1"
    field_mapper = {
      age = age
      name = new_name
    }
  }
}

sink {
  Console {
    source_table_name = "fake1"
  }
}

离线安装

下载seatunnle：https://seatunnel.apache.org/download/

解压安装包

下载所需连接器：https://repo.maven.apache.org/maven2/org/apache/seatunnel/

配置验证任务：

参照离线安装

启动seatunnel任务

cd "apache-seatunnel-${version}"
./bin/seatunnel.sh --config ./config/v2.batch.config.template -e local

TroubleShooting

1. NoClassDefFoundError: com/sun/jersey/client/impl/CopyOnWriteHashMap

2023-11-24 14:53:45,169 INFO  org.apache.seatunnel.engine.server.master.JobMaster - Init JobMaster for Job SeaTunnel_Job (780325195574411265) 
2023-11-24 14:53:45,170 INFO  org.apache.seatunnel.engine.server.master.JobMaster - Job SeaTunnel_Job (780325195574411265) needed jar urls [file:/opt/seatunnel/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar, file:/opt/seatunnel/apache-seatunnel-2.3.3/lib/seatunnel-transforms-v2.jar, file:/opt/seatunnel/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar]
2023-11-24 14:53:45,245 ERROR org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-82233] [5.1] submit job 780325195574411265 error java.lang.NoClassDefFoundError: com/sun/jersey/client/impl/CopyOnWriteHashMap
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointPlan$Builder.<init>(CheckpointPlan.java:66)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointPlan$Builder.<init>(CheckpointPlan.java:63)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointPlan.builder(CheckpointPlan.java:35)
        at org.apache.seatunnel.engine.server.dag.physical.PhysicalPlanGenerator.lambda$generate$0(PhysicalPlanGenerator.java:193)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.apache.seatunnel.engine.server.dag.physical.PhysicalPlanGenerator.generate(PhysicalPlanGenerator.java:214)
        at org.apache.seatunnel.engine.server.dag.physical.PlanUtils.fromLogicalDAG(PlanUtils.java:61)
        at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:215)
        at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$5(CoordinatorService.java:461)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.client.impl.CopyOnWriteHashMap
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)

解决：下载jar包 jersey-bundle-1.19.4.jar，复制到${seatunnel_home}/lib

2. NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream

2023-11-24 15:22:41,197 INFO  org.apache.seatunnel.engine.server.master.JobMaster - Job SeaTunnel_Job (780332477079093249) needed jar urls [file:/opt/seatunnel/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar, file:/opt/seatunnel/apache-seatunnel-2.3.3/lib/seatunnel-transforms-v2.jar, file:/opt/seatunnel/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar]
2023-11-24 15:22:41,292 ERROR org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-387863] [5.1] submit job 780332477079093249 error java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
        at org.apache.seatunnel.engine.checkpoint.storage.hdfs.common.HdfsFileStorageInstance.getOrCreateStorage(HdfsFileStorageInstance.java:53)
        at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorageFactory.create(HdfsStorageFactory.java:75)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.<init>(CheckpointManager.java:103)
        at org.apache.seatunnel.engine.server.master.JobMaster.initCheckPointManager(JobMaster.java:251)
        at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:234)
        at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$5(CoordinatorService.java:461)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 11 more
 
2023-11-24 15:22:41,298 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-387863] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN

 解决：下载jar包（seatunnel-hadoop3-3.1.4-uber-2.3.3.jar，seatunnel-hadoop3-3.1.4-uber-2.3.3-optional.jar）具体要结合自己的hadoop版本看下载哪个包，复制到${seatunnel_home}/lib