Postgresql同步数据到es需要借助中间工具连接器，连接器部署主要有两种方式，一种是基于Elastic云托管的连接器(Native connectors), 另外一种自己安装管理的连接器(self-managed connector).
托管方式连接器的使用方法文档：https://www.elastic.co/guide/en/enterprise-search/8.13/native-connectors.html
本文主要介绍如何自己安装配置连接器：
前提条件：
- Linux 或者 macOS系统
- Python 3.10或者更高版本
- 系统安装了Docker与Docker compose
我们在启动连接器的时候，要根据提示来设置连接器配置，所以必做还要安装 `build-essential` , 否则会报错
sudo apt install build-essential
下载连接器仓库源码：
git clone https://github.com/elastic/connectors.git && cd connectors
cp config.yml.example config.yml # 要先手动复制一份配置文件并命名为config.yml，否则后面运行脚本会报找不到这个文件
执行安装脚本
k3s@k3s-test-master:~/connectors$ ELASTIC_PASSWORD="my_new_password" CONNECTORS_VERSION=8.11.2.0 ./scripts/stack/run-stack.sh --no-snapshot
~/connectors/scripts/stack ~/connectors
Using compose file at: /home/k3s/connectors/scripts/stack/docker/docker-compose.yml
set PROJECT_ROOT to /home/k3s/connectors
ELASTICSEARCH_VERSION=8.11.2
KIBANA_VERSION=8.11.2
CONNECTORS_VERSION=8.11.2.0
Starting Elasticsearch...
Creating network "docker_connectors_stack_net" with the default driver
Creating docker_elasticsearch_1 ... done
Connecting to Elasticsearch on http://localhost:9200
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Starting Kibana...
docker_elasticsearch_1 is up-to-date
Creating docker_kibana_1 ... done
Connecting to Kibana on http://localhost:5601
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Updating Kibana password in Elasticsearch running on http://localhost:9200
{ }
Do you want to run the configurator? (y/n) y
ELASTICSEARCH_VERSION=8.11.2
KIBANA_VERSION=8.11.2
CONNECTORS_VERSION=8.11.2.0
config.yml already exists in /home/k3s/connectors/scripts/stack/connectors-config. Not overwriting.
ELASTICSEARCH_VERSION=8.11.2
KIBANA_VERSION=8.11.2
CONNECTORS_VERSION=8.11.2.0
脚本会调用 docker-compose安装 连接器、kibana和es，变量ELASTIC_PASSWORD指定kibana登陆密码，CONNECTORS_VERSION 指定这三个服务使用的版本（如果不使用脚本安装的话最好也要保持它们的版本一致). Kibana和ES容器启动后，会提示选择一个连接器类型，我这里使用的数据源是postgres, 使用鼠标上下键选择，enter确定
创建完成，验证：
k3s@k3s-test-master:~/connectors$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4afc1d4bb5fd docker.elastic.co/enterprise-search/elastic-connectors:8.11.2.0 "/app/bin/elastic-in…" 7 minutes ago Up 7 minutes docker_elastic-connectors_1 a0779b7bf5a6 docker.elastic.co/kibana/kibana:8.11.2 "/bin/tini -- /usr/l…" 19 minutes ago Up 19 minutes 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp docker_kibana_1 f21bf94c223b docker.elastic.co/elasticsearch/elasticsearch:8.11.2 "/bin/tini -- /usr/l…" 20 minutes ago Up 20 minutes 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp docker_elasticsearch_1
浏览器输入kibana地址：http://[服务器IP]:5601 帐号是 elastic, 密码是刚才设置的环境变量ELASTIC_PASSWORD的值。 进入后，点击左上角三条横线图标，然后Search栏Content，在右边Available indices栏就可以看到我们刚才创建的索引了
点击进入创建的索引，在Configuration配置中选择右边Sync菜单中Full Content，就开始把Postgresql数据库的相关库和表导入到es了
同步完成后页面会有提示，或者从连接器的日志也可以看出来
k3s@k3s-test-master:~/connectors$ docker logs 4afc
[FMWK][14:40:33][INFO] Running connector service version 8.11.2.0
[FMWK][14:40:33][INFO] Loading config from /config/config.yml
[FMWK][14:40:33][INFO] Running preflight checks
[FMWK][14:40:33][INFO] Waiting for NodeConfig(scheme='http', host='localhost', port=9200, path_prefix='', headers={}, connections_per_node=10, request_timeout=10.0, http_compress=False, verify_certs=True, ca_certs=None, client_cert=None, client_key=None, ssl_assert_hostname=None, ssl_assert_fingerprint=None, ssl_version=None, ssl_context=None, ssl_show_warn=True, _extras={}) (so far: 0 secs)
[FMWK][14:40:33][INFO] Extraction service is not configured, skipping its preflight check.
[FMWK][14:40:33][INFO] Job Scheduling Service started, listening to events from http://localhost:9200
[FMWK][14:40:33][INFO] Job Execution Service started, listening to events from http://localhost:9200
[FMWK][13:19:08][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Executing full sync
[FMWK][13:19:08][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Filtering validation started
[FMWK][13:19:08][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Collecting local document ids
[FMWK][13:19:08][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Iterating on remote documents
[FMWK][13:19:08][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Sync progress -- created: 0 | updated: 0 | deleted: 0
[FMWK][13:19:08][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Sync progress -- created: 0 | updated: 100 | deleted: 0
[FMWK][13:19:09][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Sync ended with status completed -- created: 0 | updated: 131 | deleted: 0 (took 1 seconds)
如果要添加索引的话，可以在 Content页面右上角"Create a new index"
选择"Use a connector"
Select a connector 页面选择你的数据源类型，点击 "Continue"
输入索引名字，然后“Continue"
点击”Generate API key" 创建一个api
把生成的新的连接配置填入连接器配置文件：
k3s@k3s-test-master:~/connectors$ cat scripts/stack/connectors-config/config.yml connectors: - api_key: eUVHa2lvNEJlbFlQeWFpLXhJcmY6S1BDTHBHZElTQkNzMVpoUzFveFJjUQ== connector_id: ykGkio4BelYPyai-xYrG service_type: postgresql - connector_id: "5EGLj44BelYPyai-lIpr" service_type: "postgresql" api_key: "NVVHTGo0NEJlbFlQeWFpLTdZclM6VEpOZEwwTWhSTm1pUDJLdGExXzdvdw==" elasticsearch.host: http://localhost:9200 elasticsearch.password: my_new_password elasticsearch.username: elastic
重启连接器容器：
k3s@k3s-test-master:~/connectors$ docker restart docker_elastic-connectors_1 docker_elastic-connectors_1
重启完成后就可以回到刚才新建的索引的 “Configuration"第4步，填入你需要连接的数据库相关信息，保存即可
我们也可以在”Scheduling"选项卡中打开自动同步，并设置同步频率.
标签：...,同步,Postgresql,Waiting,Kibana,running,Elasticsearch,connectors From： https://www.cnblogs.com/rick-zhang/p/18106088