首页 > 其他分享 >在Docker容器中运行Elasticsearch、Kinbana、Cerebo、Logstash

在Docker容器中运行Elasticsearch、Kinbana、Cerebo、Logstash

时间：2022-10-16 11:36:29浏览次数：96  
标签：es7 Kinbana name Cerebo hot warm elasticsearch Elasticsearch cold

确保自己的Centos环境中已经安装好了Docker，Docker-compose相关的软件

安装cerebro、es、kibana编写docker-compose.yml文件，部署单机环境

version: '3.5'
services:
  cerebro:
    image: lmenezes/cerebro:latest
    container_name: cerebro
    ports:
      - "9300:9300"
    command:
      - -Dhosts.0.host=http://elasticsearch:9200
    networks:
      - es7net
  kibana:
    image: kibana:7.1.0
    container_name: kibana7
    environment:
      I18N_LOCALE: "zh-CN"
      XPACK_GRAPH_ENABLED: true
      TIMELION_ENABLED: true
      XPACK_MONITORING_COLLECTION_ENABLED: true
    ports:
      - "5601:5601"
    networks:
      - es7net
  elasticsearch:
    image: elasticsearch:7.1.0
    container_name: es7
    environment:
      cluster.name: geektime
      node.name: es7
      bootstrap.memory_lock: true
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
      discovery.seed_hosts: es7
      cluster.initial_master_nodes: es7
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es7data/:/usr/share/elasticsearch/data/
    ports:
      - 9200:9200
    networks:
      - es7net

networks:
  es7net:
    driver: bridge

创建文件夹并执行脚本，下载镜像文件

mkdir -p ~/elastic/es7data
docker-compose up -d

image.png

验证是否启动成功

sudo docker images
sudo docker ps

image.png
image.png

安装cerebro、es、kibana编写docker-compose.yml文件，部署集群脚本

version: '3.5'
services:
  cerebro:
    image: lmenezes/cerebro:latest
    container_name: cerebro
    ports:
      - "9600:9000"
    command:
      - -Dhosts.0.host=http://elasticsearch:9200
    networks:
      - es7net
  kibana:
    image: kibana:7.1.0
    container_name: kibana7
    environment:
      I18N_LOCALE: "zh-CN"
      XPACK_GRAPH_ENABLED: true
      TIMELION_ENABLED: true
      XPACK_MONITORING_COLLECTION_ENABLED: true
    ports:
      - "5601:5601"
    networks:
      - es7net
  elasticsearch:
    image: elasticsearch:7.1.0
    container_name: es7_hot
    environment:
      cluster.name: geektime
      node.name: es7_hot
      node.attr.box_type: hot
      bootstrap.memory_lock: true
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
      discovery.seed_hosts: es7_hot,es7_warm,es7_cold
      cluster.initial_master_nodes: es7_hot,es7_warm,es7_cold
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es7data_hot/:/usr/share/elasticsearch/data/
    ports:
      - 9200:9200
    networks:
      - es7net
  elasticsearch2:
    image: elasticsearch:7.1.0
    container_name: es7_warm
    environment:
      cluster.name: geektime
      node.name: es7_warm
      node.attr.box_type: warm
      bootstrap.memory_lock: true
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
      discovery.seed_hosts: es7_hot,es7_warm,es7_cold
      cluster.initial_master_nodes: es7_hot,es7_warm,es7_cold
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es7data_warm/:/usr/share/elasticsearch/data/
    networks:
      - es7net
  elasticsearch3:
    image: elasticsearch:7.1.0
    container_name: es7_cold
    environment:
      cluster.name: geektime
      node.name: es7_cold
      node.attr.box_type: cold
      bootstrap.memory_lock: true
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
      discovery.seed_hosts: es7_hot,es7_warm,es7_cold
      cluster.initial_master_nodes: es7_hot,es7_warm,es7_cold
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es7data_cold/:/usr/share/elasticsearch/data/
    networks:
      - es7net

volumes:
  es7data_hot:
    driver: local
  es7data_warm:
    driver: local
  es7data_cold:
    driver: local

networks:
  es7net:
    driver: bridge

创建文件夹并执行脚本

sudo mkdir es7data_hot es7data_warm es7data_cold
sudo docker compose up -d

验证是否启动成功

image.pngimage.png
image.png

安装cerebro、es、kibana、logstash编写docker-compose.yml文件

version: '3.5'
services:
  cerebro:
    image: lmenezes/cerebro:latest
    container_name: cerebro
    ports:
      - "9600:9000"
    command:
      - -Dhosts.0.host=http://elasticsearch:9200
    networks:
      - es7net
  kibana:
    image: kibana:7.1.0
    container_name: kibana7
    environment:
      I18N_LOCALE: "zh-CN"
      XPACK_GRAPH_ENABLED: true
      TIMELION_ENABLED: true
      XPACK_MONITORING_COLLECTION_ENABLED: true
    ports:
      - "5601:5601"
    networks:
      - es7net
  elasticsearch:
    image: elasticsearch:7.1.0
    container_name: es7_hot
    environment:
      cluster.name: geektime
      node.name: es7_hot
      node.attr.box_type: hot
      bootstrap.memory_lock: true
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
      discovery.seed_hosts: es7_hot,es7_warm,es7_cold
      cluster.initial_master_nodes: es7_hot,es7_warm,es7_cold
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es7data_hot/:/usr/share/elasticsearch/data/
    ports:
      - 9200:9200
    networks:
      - es7net
  elasticsearch2:
    image: elasticsearch:7.1.0
    container_name: es7_warm
    environment:
      cluster.name: geektime
      node.name: es7_warm
      node.attr.box_type: warm
      bootstrap.memory_lock: true
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
      discovery.seed_hosts: es7_hot,es7_warm,es7_cold
      cluster.initial_master_nodes: es7_hot,es7_warm,es7_cold
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es7data_warm/:/usr/share/elasticsearch/data/
    networks:
      - es7net
  elasticsearch3:
    image: elasticsearch:7.1.0
    container_name: es7_cold
    environment:
      cluster.name: geektime
      node.name: es7_cold
      node.attr.box_type: cold
      bootstrap.memory_lock: true
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
      discovery.seed_hosts: es7_hot,es7_warm,es7_cold
      cluster.initial_master_nodes: es7_hot,es7_warm,es7_cold
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es7data_cold/:/usr/share/elasticsearch/data/
    networks:
      - es7net
  logstash:
    image: logstash:7.1.0
    container_name: logstash
    volumes:
      - ./logstash/logstash.conf:/usr/share/logstash/config/logstash.conf
      - ./logstash/data/:/usr/share/logstash/data/
    depends_on:
      - elasticsearch
      - kibana
      - elasticsearch2
      - elasticsearch3
    command: bash -c "logstash -f /usr/share/logstash/config/logstash.conf"
    ports:
      - 4560:4560
    networks:
      - es7net

volumes:
  es7data_hot:
    driver: local
  es7data_warm:
    driver: local
  es7data_cold:
    driver: local
  logstash:
    driver: local

networks:
  es7net:
    driver: bridge

创建文件夹和文件，并执行脚本

  • 创建文件夹和编写logstash启动脚本
sudo mkdir -p logstash/data
cd logstash
vi logstash.conf


input {
  file {
    path => "/usr/share/logstash/data/movies.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
    separator => ","
    columns => ["id","content","genre"]
  }

  mutate {
    split => { "genre" => "|"}
    remove_field => ["path", "host", "@timestamp","message"]
  }

  mutate {
    split => ["content", "("]
    add_field => {"title" => "%{[content][0]}"}
    add_field => {"year" => "%{[content][1]}"}
  }

  mutate {
    convert => {
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host", "@timestamp","message","content"]
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "movies"
    document_id => "%{id}"
  }
  stdout {}
}

image.png

验证是否启动成功

image.png

异常信息处理

image.png

  1. max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
  • 解决方法：修改虚拟机配置，将该值调整大
sudo vi /etc/sysctl.conf
## 追加后边的参数
vm.max_map_count=655360
## 查看是否成功
sudo sysctl -p
  1. memory locking requested for elasticsearch process but memory is not locked
  • 解决方法：在docker-compose.yml文件中配置ulimits参数信息
  elasticsearch:
    image: elasticsearch:7.1.0
    container_name: es7
    environment:
      cluster.name: geektime
      node.name: es7
      bootstrap.memory_lock: true
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
      discovery.seed_hosts: es7
      cluster.initial_master_nodes: es7
################添加如下环境变量信息################
    ulimits:
      memlock:
        soft: -1
        hard: -1
  1. the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
  • 解决方法：在docker-compose.yml文件中配置这些环境变量
  elasticsearch:
    image: elasticsearch:7.1.0
    container_name: es7
    environment:
      cluster.name: geektime
      node.name: es7
      bootstrap.memory_lock: true
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
#########添加如下环境变量信息#################
      discovery.seed_hosts: es7
      cluster.initial_master_nodes: es7

标签：es7,Kinbana,name,Cerebo,hot,warm,elasticsearch,Elasticsearch,cold
From： https://www.cnblogs.com/tenic/p/16795828.html

相关文章

  • 使用Docker容器安装Elasticsearch，Kibana
    安装前请确保linux已经安装好了Docker和DockerCompose，并配置好Docker镜像容器加速，推荐使用国内镜像加速地址，例如阿里云或者DaoCloud的镜像加速服务。需要注册帐号准备......
  • Java注解（3）：一个真实Elasticsearch案例
    学会了技术就要使用，否则很容易忘记，因为自然界压根就不存在什么代码、变量之类的玩意，这都是一些和生活常识格格不入的东西。只能多用多练，形成肌肉记忆才行。在一次实际的产......
  • Elasticsearch 和 Kibana 安装
    Elasticsearch和Kibana安装操作系统：Windows10ElasticStack版本：8.4.3目录Elasticsearch和Kibana安装安装Elasticsearch客户端连接Elasticsearch安装Kibana......
  • elasticsearch的查询数大于10000查询失败
    错误信息Causedby:ElasticsearchException[Elasticsearchexception[type=illegal_argument_exception,reason=Resultwindowistoolarge,from+sizemustbeles......
  • ElasticSearch
    选型因为数据库的like等操作会导致索引失效，进而导致查询性能低，功能弱，从而引入ElasticSearch通过倒排索引解决全文检索性能和功能的问题。数据结构倒排索引一般包含两......
  • Elasticsearch使用terms聚合之后进行分页排序
    引言elasticsearch中实现聚合也非常常见，同时es的数据量一般比较大，因此聚合结果比较多，像terms聚合默认只返回10条聚合结果，所以聚合之后进行分页，也是非常常见的操作。es的t......
  • Elasticsearch——Query DSL语法入门
    QueryDSL入门官网介绍链接： https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.htmlsearchapi就是对存储在elasticsearch（以下简称es）中的数......
  • Elasticsearch——java api构建搜索
    版本不同版本的elasticsearch-rest-high-level-client和elasticsearch之间存在兼容风险，请确保和elasticsearch版本一致，否则会出现无法预计的错误。es配置maven依赖<dep......
  • Elasticsearch——评分机制详解
    前言一个搜索引擎使用的时候必定需要排序这个模块，如果在不选择按照某一字段排序的情况下，都是按照打分的高低进行一个默认排序的，所以如果正式使用的话，必须对默认排序的打分......
  • Elasticsearch——聚合搜索
    聚合分析简介聚合分析：英文为Aggregation，是es除搜索功能外提供的针对es数据做统计分析的功能。功能丰富，提供Bucket、Metric、Pipeline等多种分析方式，可以满足大部分的分析......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99