flume使用教程

1.

启动端口

   nc node101 55555

监听端口

   nc -l 55555

job下配置文件

# example.conf: A single-node Flume configuration

# Name the components on this agent

a1.sources = r1

a1.sinks = k1

a1.channels = c1

# Describe/configure the source

a1.sources.r1.type = netcat

a1.sources.r1.bind = localhost 

a1.sources.r1.port = 44444

# Describe the sink

a1.sinks.k1.type = logger

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

启动服务

bin/flume-ng agent --conf conf --conf-file job/log_to_console.conf --name a1

控制台打印

bin/flume-ng agent --conf conf --conf-file job/log_to_console.conf --name a1 -Dflume.root.logger=INFO,console

杀死进程 kill -9  端口号

数据生成

创建日志文件

Hello.log

创建生成器

Write.sh

运行生成器  sh write.sh

查看数据文件：tail -F hello.txt

控制太配置

修改配置

a1.sources.r1.type = exec

a1.sources.r1.command = tail -F /var/log/secure

增加配置 hdfs配置

a1.channels = c1

a1.sinks = k1

a1.sinks.k1.type = hdfs

a1.sinks.k1.channel = c1

a1.sinks.k1.hdfs.path = hdfs:node101:8020/flume/events/%Y-%m-%d/%H%M/%S

a1.sinks.k1.hdfs.filePrefix = events-

a1.sinks.k1.hdfs.round = true

a1.sinks.k1.hdfs.roundValue = 10

a1.sinks.k1.hdfs.roundUnit = minute

1. 写日志文件
2. 写数据生成器
3. 写flume配置文件
4. 启动hdp.sh hdfs
5. 启动hdp.sh yarn
6. 启动flume