首页 > 系统相关 >hello flume (Ubuntu 下 flume1.5单机版安装以及简单入门示例)

hello flume (Ubuntu 下 flume1.5单机版安装以及简单入门示例)

时间：2023-04-25 19:01:39浏览次数：45  
标签：flume INFO 13 单机版 示例 java apache org


1,下载最新的flume安装包：

wget http://www.apache.org/dist/flume/stable/apache-flume-1.5.2-bin.tar.gz





2，在安装目录解压：

tar -zxvf apache-flume-1.5.2-bin.tar.gz





3,设置环境变量

export JAVA_HOME=/usr ;
export FLUME_HOME=/home/joeyon/apache-flume-1.5.2-bin;
export PATH=$PATH:FLUME_HOME/bin



4,配置一个简单单机的测试案例 


</pre>4.1在conf目录下新建一个文件，example-conf.properties，内容如下：<pre name="code" class="html"># Describe the source 
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
# 将数据输出至日志中
a1.sinks.k1.type = logger

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1




  4.2在conf目录下新建一个源文件file01并向其中写入数据

# Use a channel which buffers events in memory
   touch file01;

   echo "hello world" > file01



   4.3 启动agent代理 

flume-ng agent -c . -f /home/joeyon/apache-flume-1.5.2-bin/conf/example-conf.properties  -n a1 -Dflume.root.logger=INFO,console

启动成功日志如下： 

+ exec /usr/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/data/projects/apache-flume-1.5.2-bin/conf:/data/projects/apache-flume-1.5.2-bin/lib/*' -Djava.library.path= org.apache.flume.node.Application -f /data/projects/apache-flume-1.5.2-bin/conf/example-conf.properties -n a1
2015-01-21 13:44:43,231 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2015-01-21 13:44:43,239 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:/data/projects/apache-flume-1.5.2-bin/conf/example-conf.properties
2015-01-21 13:44:43,248 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: k1 Agent: a1
2015-01-21 13:44:43,249 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2015-01-21 13:44:43,249 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2015-01-21 13:44:43,261 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [a1]
2015-01-21 13:44:43,262 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:150)] Creating channels
2015-01-21 13:44:43,271 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:40)] Creating instance of channel c1 type memory
2015-01-21 13:44:43,276 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel c1
2015-01-21 13:44:43,277 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:39)] Creating instance of source r1, type avro
2015-01-21 13:44:43,297 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: k1, type: logger
2015-01-21 13:44:43,301 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)] Channel c1 connected to [r1, k1]
2015-01-21 13:44:43,309 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:Avro source r1: { bindAddress: localhost, port: 44444 } }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@27b6a3ac counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
2015-01-21 13:44:43,320 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel c1
2015-01-21 13:44:43,383 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
2015-01-21 13:44:43,383 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started
2015-01-21 13:44:43,387 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink k1
2015-01-21 13:44:43,388 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source r1
2015-01-21 13:44:43,388 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:228)] Starting Avro source r1: { bindAddress: localhost, port: 44444 }...
2015-01-21 13:44:43,771 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
2015-01-21 13:44:43,771 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: r1 started
2015-01-21 13:44:43,772 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:253)] Avro source r1 started.



4.4.启动avro-client客户端向agent代理发送数据： 

flume-ng avro-client -c . -H localhost -p 44444 -F /home/joeyon/apache-flume-1.5.2-bin/conf/file01


日志如下： 

+ exec /usr/bin/java -Xmx20m -cp '/data/projects/apache-flume-1.5.2-bin/conf:/data/projects/apache-flume-1.5.2-bin/lib/*' -Djava.library.path= org.apache.flume.client.avro.AvroCLIClient -H localhost -p 44444 -F /home/joeyon/apache-flume-1.5.2-bin/conf/file01




4.5查看agen控制台输出代理端输出如下则收集成功：

2015-01-21 13:50:13,276 (New I/O server boss #1 ([id: 0xc0b7fffb, /127.0.0.1:44444])) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 => /127.0.0.1:44444] OPEN
2015-01-21 13:50:13,277 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 => /127.0.0.1:44444] BOUND: /127.0.0.1:44444
2015-01-21 13:50:13,278 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 => /127.0.0.1:44444] CONNECTED: /127.0.0.1:3809
2015-01-21 13:50:13,703 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 :> /127.0.0.1:44444] DISCONNECTED
2015-01-21 13:50:13,703 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 :> /127.0.0.1:44444] UNBOUND
2015-01-21 13:50:13,703 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 :> /127.0.0.1:44444] CLOSED
2015-01-21 13:50:13,704 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed(NettyServer.java:209)] Connection to /127.0.0.1:3809 disconnected.
2015-01-21 13:50:17,411 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 3E 3E 3E 3E 3E hello world>>>>> }
2015-01-21 13:50:17,411 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 6D 79 20 6E 61 6D 65 20 69 73 20 6A 6F 65 79 6F my name is joeyo }
2015-01-21 13:50:17,412 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 77 65 6C 63 6F 6D 20 74 6F 20 63 68 69 6E 61 3E welcom to china> }




标签：flume,INFO,13,单机版,示例,java,apache,org
From： https://blog.51cto.com/u_16088628/6224885

相关文章

  • jenkins 用法示例
    示例1：根据参数进行构建  根据参数的值进行不同的操作。#!/bin/bashecho"获取值"echo$Deploycase$Deployin"deploy1")echo123>>/opt/script/aa.log;;"rollback1")echo456>>/opt/script/aa.log;;esac ......
  • Linux基础知识（17）- Kerberos (二) | krb5 API 的 C 程序示例
    在“Linux基础知识（16）-Kerberos(一)|Kerberos安装配置”里我们演示了Kerberos安装配置和Kadmin等命令行工具的用法，本文将演示krb5API的使用方法。Krb5API:http://web.mit.edu/kerberos/krb5-current/doc/appldev/refs/api/index.html 1.系统环境   操作......
  • Watch监听-示例
    packagecom.hw.curator;importorg.apache.curator.RetryPolicy;importorg.apache.curator.framework.CuratorFramework;importorg.apache.curator.framework.CuratorFrameworkFactory;importorg.apache.curator.framework.api.BackgroundCallback;importorg.apach......
  • python三角网格划分示例
    python三角网格划分示例 importnumpyasnpimportturtle#输入三角形的边长length=float(input("Enterthelengthofthetriangle:"))#计算最短边、最长边和三角形个数short_side=lengthmax_side=length+lengthn=int(max_side/s......
  • 正的浮点数相加的示例程序 Verilog
    modulefloat_adder( input        clk, input       rst_n, input        en, input   [31:0]  aIn, input   [31:0]  bIn, outputreg     busy, outputreg   out......
  • 设计模式-原型模式-Java中使用示例-节日发送邮件活动
    场景设计模式-原型模式-浅克隆和深克隆在Java中的使用示例：https://blog.csdn.net/BADAO_LIUMANG_QIZHI/article/details/127576328上面初步记录了原型模式的使用示例，下面再记录一个银行节假日或者搞活动时发送邮件的例子。原型模式原型模式（PrototypePattern）的简单程度仅次......
  • golang中sync.Pool的使用示例
    先上代码：packagemainimport( "fmt" "sync")varpoolsync.Pooltypepersonstruct{ Namestring Ageint}funcinit(){ pool=sync.Pool{New:func()any{ returnnew(person) }}}funcmain(){ p:=pool.Get().(*person) p......
  • Google Guava常用的代码示例
    GoogleGuava谷歌出品的，非常实用。包含集合、并发、I/O、散列、缓存、字符串等。依赖：<dependency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <version>18.0</version></dependency>JoinerJoiner可以连接字符串。常用方法：Joine......
  • MKL稀疏矩阵运算示例及函数封装
    IntelMKL库提供了大量优化程度高、效率快的稀疏矩阵算法，使用MKL库的将大型矩阵进行稀疏表示后，利用稀疏矩阵运算可大量节省计算时间和空间，但由于MKL中的原生API接口繁杂，因此将常用函数封装，便于后续使用，最后在实际例子中调用接口执行想要的矩阵运算。0稀疏矩阵稀疏矩阵是指矩阵......
  • thinkphp在模型中自动完成session赋值示例代码
    相信用过thinkphp的用户都知道thinkphp的模型可以完成很多辅助功能，比如自动验证、自动完成等，今天在开发中遇到自动完成中需要获取session值然后自动赋值的功能，具体看代码；classArticlelModelextendsModel{protected$_auto=array(array('addtime','time',1,'function'......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99