利器 | 测试必会之 Linux 三剑客 ( grep / awk / sed )
IT牛客 IT牛客 2023-05-02 21:00 发表于河南 收录于合集 #linux13个 #awk1个 #sed1个原文:
blog.csdn.net/weixin_43291944/article/details/100180464 IT牛客 专注于IT技术分享,原创技术和创新技术分享,IT牛客,我们一起进步…… 12篇原创内容 公众号
1. grep
grep-global regular expression print - 全局正则表达式打印
可用于数据查找定位
curl -s https://testerhome.com | grep href
curl -s https://testerhome.com | grep href | grep -o "http[^\"]*"
curl -s -I https://testerhome.com/topics/feed
curl -s https://testerhome.com | grep href | grep -o "http[^\"]*" | while read line;do curl -s -I $line | grep 200 && echo $line || echo ERR $line;done
4. 最终结果展示
2. awk
awk的字段数据处理
-
-F 参数指定字段分隔符
-
BEGIN{FS=‘_’} 也可以表示分隔符
$0 代表原来的行
$1 代表第一个字段
$N 代表第N个字段
$NF 代表最后一个字段
下面以一个在nginx.log中查找返回状态码非200的请求响应数目的需求为例,演示awk的基础用法
有一份nginx.log文件,打开后内容格式如下:
220.181.108.111 - - [05/Dec/2018:00:11:42 +0000] "GET /topics/15225/show_wechat HTTP/1.1" 200 1684 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 0.029 0.029 .
216.244.66.241 - - [05/Dec/2018:00:11:42 +0000] "GET /topics/10052/replies/85845/reply_suggest HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.016 0.016 .
216.244.66.241 - - [05/Dec/2018:00:11:42 +0000] "GET /topics/10040?order_by=created_at HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.002 0.002 .
216.244.66.241 - - [05/Dec/2018:00:11:42 +0000] "GET /topics/10043/replies/85544/reply_suggest HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.001 0.001 .
216.244.66.241 - - [05/Dec/2018:00:11:44 +0000] "GET /topics/10075/replies/89029/edit HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.001 0.001 .
216.244.66.241 - - [05/Dec/2018:00:11:44 +0000] "GET /topics/10075/replies/89631/edit HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.001 0.001 .
216.244.66.241 - - [05/Dec/2018:00:11:45 +0000] "GET /topics/10075?order_by=created_at HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.000 0.000 .
216.244.66.241 - - [05/Dec/2018:00:11:45 +0000] "GET /topics/10075?order_by=like HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.001 0.001 .
223.71.41.98 - - [05/Dec/2018:00:11:46 +0000] "GET /cable HTTP/1.1" 101 60749 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0" 2608.898 2608.898 .
113.87.161.17 - - [05/Dec/2018:00:11:39 +0000] "GET /cable HTTP/1.1" 101 3038 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36" 112.418 112.418 .
216.244.66.241 - - [05/Dec/2018:00:11:46 +0000] "GET /topics/10079/replies/119591/edit HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.001 0.001 .
216.244.66.241 - - [05/Dec/2018:00:11:46 +0000] "GET /topics/10089?locale=zh-TW HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.002 0.002 .
观察log内容,可以发现,以空格为分隔符,状态码在第九个字段位置;这里我们用awk命令从第九个字段位置开始匹配非200的状态码并打印出来。命令:
awk '$9!~/200/{print $9}' nginx.log
[avbxb9efockpz ~]$ awk '$9!~/200/{print $9}' nginx.log301
301
301
301
301
301
301
301
301
......#剩余部分省略
再对取出的数据进行排序->去重->按数字的倒叙进行排列。命令:
awk '$9!~/200/{print $9}' nginx.log | sort | uniq -c | sort -nr
命令含义:
sort: 按从小到大进行排序
uniq -c :去重(相邻)
-nr: 按数字进行倒叙排序
-n:按数字进行排序
结果展示:
[sqavbxb9efockpz ~]$ awk '$9!~/200/{print $9}' nginx.log | sort | uniq -c | sort -nr
433 101
304 301
266 404
152 302
7 401
5 304
2 499
2 422
1 500
1 nobody2 root
3 daemon
4 _uucp
5 _taskgated
6 _networkd
7 _installassistant
8 _lp
9 _postfix
......
用户信息:
cat /etc/passwd | awk -F ':' 'BEGINE{userindex=0}{user[userindex]=$1;userindex++}END{for(i=0;i<NR;i++)print i+1, user[i+10]}' |less
3. sed
sed:stream editor 根据定位到的数据行修改数据
[16210504@izuf60jasqavbxb9efockpz ~]$ echo "aaa|bbb}|cccbbb" | sed 's/bbb/BBB/'
aaa|BBB}|cccbbb
[16210504@izuf60jasqavbxb9efockpz ~]$ echo "aaa|bbb}|cccbbb" | sed 's#bbb#BBB#'
aaa|BBB}|cccbbb
-
若想讲目标中所有的字段都替换,需要在命令最后加上g:
[16210504@izuf60jasqavbxb9efockpz ~]$ echo "aaa|bbb}|cccbbb" | sed 's/bbb/BBB/g'
aaa|BBB}|cccBBB
-
sed还可以修改文件中的内容,现在有文件text.txt,内容如下:
[16210504@izuf60jasqavbxb9efockpz ~]$ cat text.txt
hello bash world
hi~ tester
go go go go!
用sed 's/hello/HELLO/' text.txt
命令将文件中的hello
替换成HELLO
:
[16210504@izuf60jasqavbxb9efockpz ~]$ sed 's/hello/HELLO/' text.txt
HELLO bash world
hi~ tester
go go go go!
但是此时我们打开源text.txt文件发下源文件内容并未改变:
[16210504@izuf60jasqavbxb9efockpz ~]$ cat text.txt
hello bash world
hi~ tester
go go go go!
[16210504@izuf60jasqavbxb9efockpz ~]$ sed -i.bak 's/hello/HELLO/' text.txt
[16210504@izuf60jasqavbxb9efockpz ~]$ ls
1 1.sh Allen_qin nginx.log test text.txt text.txt.bak while_test
[16210504@izuf60jasqavbxb9efockpz ~]$ cat text.txt
HELLO bash world
hi~ tester
go go go go!
[16210504@izuf60jasqavbxb9efockpz ~]$ cat text.txt.bak
hello bash world
hi~ tester
go go go go!
a:
b:
c:
d:
要将其中每行末尾的:
都替换成@
,将a
替换成A
,并在文本末尾加上“Sed Test
”
命令:
sed -i -e 's/:/@/g' \
-i -e 's/a/A/' \
-i -e '$a Sed Test' 1.txt
实例演示:
IT牛客 专注于IT技术分享,原创技术和创新技术分享,IT牛客,我们一起进步…… 12篇原创内容 公众号[16210504@izuf60jasqavbxb9efockpz ~]$ sed -i -e 's/:/@/g' -i -e 's/a/A/' -i -e '$a Sed Test' 1.txt
[16210504@izuf60jasqavbxb9efockpz ~]$ cat 1.txt
A@
b@
c@
d@
Sed Test
如喜欢本文,请点击右上角,把文章分享到朋友圈
因公众号更改推送规则,请点“在看”并加“星标”第一时间获取精彩技术分享
·END·
收录于合集 #linux 13个 上一篇Linux命令之精确控制文件访问权限setfacl 阅读 176 IT牛客 12篇原创内容 写下你的留言 标签:00,grep,1.1,05,sed,go,com,三剑客 From: https://www.cnblogs.com/cherishthepresent/p/17446303.html