首页 > 其他分享 >Elasticsearch 入门:logstash 5.0.0 安装及输出数据到 elasticsearch

Elasticsearch 入门:logstash 5.0.0 安装及输出数据到 elasticsearch

时间:2023-01-10 12:07:26浏览次数:70  
标签:5.0 INFO pipeline 01 elasticsearch Elasticsearch 2017 logstash


首先安装:elasticsearch 、 kibana ,以下测试会用到。

安装参考:

​​Elasticsearch 入门:CentOS 5.6 安装 Elasticsearch 5.0​​

​​Elasticsearch 入门:Elasticsearch 5.0 安装 kibana 5.0​​


logstash多种安装方法: ​​https://www.elastic.co/guide/en/logstash/5.0/installing-logstash.html​

logstash下载 :​​https://www.elastic.co/downloads/logstash​

logstash5.0.0下载:​​https://www.elastic.co/downloads/past-releases/logstash-5-0-0​


部署,使用的是离线安装,因有的服务器不能连接网络

shell> tar zxvf logstash-5.0.0.tar.gz
shell> mv logstash-5.0.0 /usr/local/elasticsearch/files/logstash
shell> cd /usr/local/elasticsearch/files/logstash

运行

shell> bin/logstash -e 'input { stdin {} } output { stdout {} }'
Sending Logstash logs to /usr/local/elasticsearch/files/logstash/logs which is now configured via log4j2.properties.
The stdin plugin is now waiting for input:
[2017-01-14T20:27:54,232][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
[2017-01-14T20:27:54,429][INFO ][logstash.pipeline ] Pipeline main started
[2017-01-14T20:27:54,603][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
2017-01-14T12:27:54.427Z 0.0.0.0



在当前窗口输入第一行信息,可看到第二行输出结果:

hello logstash!
2017-01-14T12:31:14.798Z 0.0.0.0 hello logstash!


ctrl + C 退出,使用配置文件方法启动,创建配置文件(另一种格式输出:codec => rubydebug)

shell> vi config/logstashtest.conf
input {
stdin {}
}
output {
stdout {
codec => rubydebug
}
}


运行,使用配置文件

shell> bin/logstash -f config/logstashtest.conf

Sending Logstash logs to /usr/local/elasticsearch/files/logstash/logs which is now configured via log4j2.properties.
The stdin plugin is now waiting for input:
[2017-01-14T22:26:15,834][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
[2017-01-14T22:26:15,926][INFO ][logstash.pipeline ] Pipeline main started
[2017-01-14T22:26:16,404][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9601}
{
"@timestamp" => 2017-01-14T14:26:15.952Z,
"@version" => "1",
"host" => "0.0.0.0",
"message" => ""
}


输入第一行信息,可看到输出结果:

hello logstash!
{
"@timestamp" => 2017-01-14T14:26:47.163Z,
"@version" => "1",
"host" => "0.0.0.0",
"message" => "hello logstash!"
}



现在配置输出到 Elasticsearch,也保留 stdout 的输出.(注意名称:hosts!)

hell> vi config/logstashtest.conf
input {
stdin {}
}
output{
elasticsearch {
hosts => ["192.168.1.222:9200"]
index => "test"
}
stdout {
codec => rubydebug
}
}


运行,使用配置文件

shell> bin/logstash -f config/logstashtest.conf

Sending Logstash logs to /usr/local/elasticsearch/files/logstash/logs which is now configured via log4j2.properties.
The stdin plugin is now waiting for input:
[2017-01-14T23:10:11,512][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>["http://192.168.1.222:9200"]}}
[2017-01-14T23:10:11,521][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-01-14T23:10:11,785][INFO ][logstash.outputs.elasticsearch] Attempting to install template
{:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}
, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}
, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string"
, "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string"
, "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}]
, "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}
, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}
, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-01-14T23:10:17,071][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2017-01-14T23:10:33,701][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["192.168.1.222:9200"]}
[2017-01-14T23:10:33,714][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
[2017-01-14T23:10:33,715][INFO ][logstash.pipeline ] Pipeline main started
[2017-01-14T23:10:35,171][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9601}


启动使用了模板,接着输入“hello logstash!”会正常输出 ,但是有没有保存到  elasticsearch 呢??


打开浏览器访问 kibana 地址: http://192.168.1.222:5601/

点击选项 “Dev Tools”,查询输入的语句或单词:

GET _search
{
"query": {
"match_phrase": {
"message": "hello logstash!"
}
}
}

输出结果如下:

{
"took": 48,
"timed_out": false,
"_shards": {
"total": 31,
"successful": 31,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.51623213,
"hits": [
{
"_index": "test",
"_type": "logs",
"_id": "AVmdjOMfxMlXNHIPQXkG",
"_score": 0.51623213,
"_source": {
"@timestamp": "2017-01-14T15:16:05.916Z",
"@version": "1",
"host": "0.0.0.0",
"message": "hello logstash!"
}
}
]
}
}


可以看到索引为 “test” , message 为 "hello logstash!" ,已经保持进来了!

Elasticsearch 入门:logstash 5.0.0 安装及输出数据到 elasticsearch_elasticsearch

要方便在可视化界面中搜索查询, 先在 kibana 中创建一个索引,名称为 “test”

Elasticsearch 入门:logstash 5.0.0 安装及输出数据到 elasticsearch_配置文件_02

现在可以在 kibana 中查询了!

Elasticsearch 入门:logstash 5.0.0 安装及输出数据到 elasticsearch_elasticsearch_03

好了,简单测试完成!


标签:5.0,INFO,pipeline,01,elasticsearch,Elasticsearch,2017,logstash
From: https://blog.51cto.com/hzc2012/6000160

相关文章

  • ElasticSearch的基本使用
    1、ElasticSearch基本介绍TheElasticStack,包括Elasticsearch、Kibana、Beats和Logstash(也称为ELKStack)。能够安全可靠地获取任何来源、任何格式的数据,然后实时地......
  • ElasticSearch集群搭建
    一、调整内核参数vim/etc/sysctl.confvm.max_map_count=262144vim/etc/security/limits.confsoftnofile65536hardnofile65536sysctl-p二、解压文件tar-z......
  • ThinkPHP v5.0.24 反序列化
    ThinkPHPv5.0.24反序列化前言昨天花了一下午的时间才把反序列化链给审明白,今天记录一下笔记再来审一遍。(自己还是太菜了~~~)在我的印象中,ThinkPHP框架的漏洞非常多,所以......
  • Java中Elasticsearch 实现分页方式(三种方式)
    目录ES简介ES的特点:一、from+size浅分页二、scroll深分页scroll删除三、search_after深分页ES简介Elasticsearch是一个基于Lucene实现的......
  • 重学ElasticSearch (ES) :ELK搭建SpringBoot日志实时分析系统
    一、概述在一个大型的分布式架构的项目里,不同的服务模块部署在不同的服务器上,如果想要定位问题,可能需要去不同的服务器上查看不同服务的日志。那么,ELK可以很方便的把日志集......
  • ubuntu 安装elasticsearch
    elasticsearch简介  环境准备 elasticsearch:7.0.0 kibana     :7.0.0 安装 1.新创建普通用户 elasticsearch不能用root账号启动,为了避免之后......
  • windows下springboot项目部署elk日志系统教程elasticsearch与logstash与kibana
    1.项目中加入依赖:compile'net.logstash.logback:logstash-logback-encoder:6.0'如果是maven项目的话:字符串中的冒号为隔断,第一个为groupid,第二个为artifactId,第三个为versi......
  • (四)elasticsearch 源码之索引流程分析
    1.概览前面我们讨论了es是如何启动,本文研究下es是如何索引文档的。下面是启动流程图,我们按照流程图的顺序依次描述。  其中主要类的关系如下:  2.索引流程......
  • ESP32 IDF V5.0 编译环境
    方法:环境搭建工具一键安装:下载链接:https://dl.espressif.com/dl/esp-idf/  可以选择离线安装方式和在线安装方式,建议:采用离线安装的方式下载离线安装包之后点击安装......
  • elasticsearch之单请求多查询
    一、需要解决的问题有的时候我们需要同时执行多个查询,并且需要得到每个单独查询的搜索结果,elasticsearch提供了multisearch此需求的支持;二、elasticsearchmultisearch......