kestra 官方是提供了基于docker-compose 运行的模版，以下主要是体验下，实际上kestra 提供的一些能力还是很强大的
尤其是强大的插件开发能力
环境准备
- docker-compose 文件
volumes:
postgres-data:
driver: local
kestra-data:
driver: local
services:
postgres:
image: postgres
volumes:
- postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_DB: kestra
POSTGRES_USER: kestra
POSTGRES_PASSWORD: k3str4
healthcheck:
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
interval: 30s
timeout: 10s
retries: 10
kestra:
image: kestra/kestra:latest-full
pull_policy: always
entrypoint: /bin/bash
# Note that this is meant for development only. Refer to the documentation for production deployments of Kestra which runs without a root user.
user: "root"
command:
- -c
- /app/kestra server standalone --worker-thread=128
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/kestra-wd:/tmp/kestra-wd
environment:
KESTRA_CONFIGURATION: |
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driverClassName: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
server:
basic-auth:
enabled: false
username: admin
password: kestra
repository:
type: postgres
storage:
type: local
local:
base-path: "/app/storage"
queue:
type: postgres
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
url: http://localhost:8080/
ports:
- "8080:8080"
- "8081:8081"
depends_on:
postgres:
condition: service_started
运行&使用
- 启动
docker-compose up -d
- 访问
模式登陆完成，开启任务官方也有一个引导可以帮助我们快速使用kestra（会自动创建相关的组建任务）
效果如下
示例task 定义,可以看到就是一个yaml 文件，包含了输入，任务，以及触发
# Flow declaration with a mandatory unique identifier, a namespace, and an optional description.
# Flow identifier are unique inside a namespace.
id: welcomeKestra
namespace: io.kestra.tour
description: Welcome to Kestra!
# Flow inputs: each input has a name, a type, and an optional default value.
inputs:
# We define one input of name 'user' with a default value 'Data Engineer'
- name: user
type: STRING
defaults: Kestra user
# List of tasks that will be executed one after the other.
# Each task must have an identifier unique for the flow and a type.
# Depending on the type of the task, you may have to pass additional attributes.
tasks:
# This is one of the simplest task: it echos a message in the log, like the 'echo' command.
# The message is passed thanks to the 'format' attribute.
# We use the variable from the 'inputs' : {{ and }} are separator of a Pebble expression in which we can access variables.
- id: hello
type: io.kestra.core.tasks.log.Log
message: Hey there, {{ inputs.user }}!
# This task runs a bash command.
# Here we just 'echo' a message, using again the input variable.
- id: goodbye
type: io.kestra.core.tasks.scripts.Bash
commands:
- echo See you soon, {{ inputs.user }}!
# To trigger the Flow we use the 'triggers' property.
triggers:
# Here we use the 'schedule' trigger to run the flow every minute.
- id: everyMinute
type: io.kestra.core.models.triggers.types.Schedule
cron: "*/1 * * * *"
inputs:
name: Kestra master user
执行信息
说明
kestra 是基于java 开发的，开发框架基于了micronaut，支持丰富的插件开发，同时官方已经提供了不少插件(开发的插件对于包含以来的需要打包为shadow fat jar)，对于数据开发是一个不错的选择，当然kestra 是包含企业版的，社区版缺少ha 以及访问控制的能力
参考资料
https://github.com/kestra-io/kestra
https://kestra.io/docs/architecture
https://kestra.io/
https://kestra.io/enterprise