利器 | 测试必会之 Linux 三剑客 ( grep / awk / sed )
IT牛客 IT牛客 2023-05-02 21:00 发表于河南 收录于合集 #linux13个 #awk1个 #sed1个原文：
blog.csdn.net/weixin_43291944/article/details/100180464
1. grep
grep-global regular expression print - 全局正则表达式打印
可用于数据查找定位
curl -s https://testerhome.com | grep hrefcurl -s https://testerhome.com | grep href | grep -o "http[^\"]*"curl -s -I https://testerhome.com/topics/feedcurl -s https://testerhome.com | grep href | grep -o "http[^\"]*" | while read line;do curl -s -I $line | grep 200 && echo $line || echo ERR $line;done
4. 最终结果展示
2. awk
awk的字段数据处理
-
-F 参数指定字段分隔符
-
BEGIN{FS=‘_’} 也可以表示分隔符
$0 代表原来的行$1 代表第一个字段$N 代表第N个字段$NF 代表最后一个字段
下面以一个在nginx.log中查找返回状态码非200的请求响应数目的需求为例，演示awk的基础用法
有一份nginx.log文件，打开后内容格式如下：
220.181.108.111 - - [05/Dec/2018:00:11:42 +0000] "GET /topics/15225/show_wechat HTTP/1.1" 200 1684 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 0.029 0.029 .216.244.66.241 - - [05/Dec/2018:00:11:42 +0000] "GET /topics/10052/replies/85845/reply_suggest HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.016 0.016 .216.244.66.241 - - [05/Dec/2018:00:11:42 +0000] "GET /topics/10040?order_by=created_at HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.002 0.002 .216.244.66.241 - - [05/Dec/2018:00:11:42 +0000] "GET /topics/10043/replies/85544/reply_suggest HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.001 0.001 .216.244.66.241 - - [05/Dec/2018:00:11:44 +0000] "GET /topics/10075/replies/89029/edit HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.001 0.001 .216.244.66.241 - - [05/Dec/2018:00:11:44 +0000] "GET /topics/10075/replies/89631/edit HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.001 0.001 .216.244.66.241 - - [05/Dec/2018:00:11:45 +0000] "GET /topics/10075?order_by=created_at HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.000 0.000 .216.244.66.241 - - [05/Dec/2018:00:11:45 +0000] "GET /topics/10075?order_by=like HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.001 0.001 .223.71.41.98 - - [05/Dec/2018:00:11:46 +0000] "GET /cable HTTP/1.1" 101 60749 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0" 2608.898 2608.898 .113.87.161.17 - - [05/Dec/2018:00:11:39 +0000] "GET /cable HTTP/1.1" 101 3038 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36" 112.418 112.418 .216.244.66.241 - - [05/Dec/2018:00:11:46 +0000] "GET /topics/10079/replies/119591/edit HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.001 0.001 .216.244.66.241 - - [05/Dec/2018:00:11:46 +0000] "GET /topics/10089?locale=zh-TW HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])" 0.002 0.002 .
观察log内容，可以发现，以空格为分隔符，状态码在第九个字段位置；这里我们用awk命令从第九个字段位置开始匹配非200的状态码并打印出来。命令：
awk '$9!~/200/{print $9}' nginx.log[avbxb9efockpz ~]$ awk '$9!~/200/{print $9}' nginx.log301301301301301301301301301......#剩余部分省略
再对取出的数据进行排序->去重->按数字的倒叙进行排列。命令：
awk '$9!~/200/{print $9}' nginx.log | sort | uniq -c | sort -nr
命令含义：
sort: 按从小到大进行排序uniq -c :去重（相邻）-nr: 按数字进行倒叙排序-n:按数字进行排序
结果展示：
[sqavbxb9efockpz ~]$ awk '$9!~/200/{print $9}' nginx.log | sort | uniq -c | sort -nr433 101304 301266 404152 3027 4015 3042 4992 4221 500
1 nobody2 root3 daemon4 _uucp5 _taskgated6 _networkd7 _installassistant8 _lp9 _postfix......
用户信息：
cat /etc/passwd | awk -F ':' 'BEGINE{userindex=0}{user[userindex]=$1;userindex++}END{for(i=0;i<NR;i++)print i+1, user[i+10]}' |less3. sed
sed:stream editor 根据定位到的数据行修改数据
[16210504@izuf60jasqavbxb9efockpz ~]$ echo "aaa|bbb}|cccbbb" | sed 's/bbb/BBB/'aaa|BBB}|cccbbb[16210504@izuf60jasqavbxb9efockpz ~]$ echo "aaa|bbb}|cccbbb" | sed 's#bbb#BBB#'aaa|BBB}|cccbbb
-
若想讲目标中所有的字段都替换，需要在命令最后加上g:
[16210504@izuf60jasqavbxb9efockpz ~]$ echo "aaa|bbb}|cccbbb" | sed 's/bbb/BBB/g'aaa|BBB}|cccBBB
-
sed还可以修改文件中的内容，现在有文件text.txt,内容如下：
[16210504@izuf60jasqavbxb9efockpz ~]$ cat text.txthello bash worldhi~ testergo go go go!
用sed 's/hello/HELLO/' text.txt 命令将文件中的hello替换成HELLO：
[16210504@izuf60jasqavbxb9efockpz ~]$ sed 's/hello/HELLO/' text.txtHELLO bash worldhi~ testergo go go go!
但是此时我们打开源text.txt文件发下源文件内容并未改变：
[16210504@izuf60jasqavbxb9efockpz ~]$ cat text.txthello bash worldhi~ testergo go go go!
[16210504@izuf60jasqavbxb9efockpz ~]$ sed -i.bak 's/hello/HELLO/' text.txt[16210504@izuf60jasqavbxb9efockpz ~]$ ls1 1.sh Allen_qin nginx.log test text.txt text.txt.bak while_test[16210504@izuf60jasqavbxb9efockpz ~]$ cat text.txtHELLO bash worldhi~ testergo go go go![16210504@izuf60jasqavbxb9efockpz ~]$ cat text.txt.bakhello bash worldhi~ testergo go go go!
a:b:c:d:
要将其中每行末尾的：都替换成@，将a替换成A，并在文本末尾加上“Sed Test”
命令：
sed -i -e 's/:/@/g' \-i -e 's/a/A/' \-i -e '$a Sed Test' 1.txt
实例演示：
[16210504@izuf60jasqavbxb9efockpz ~]$ sed -i -e 's/:/@/g' -i -e 's/a/A/' -i -e '$a Sed Test' 1.txt[16210504@izuf60jasqavbxb9efockpz ~]$ cat 1.txtA@b@c@d@Sed Test
如喜欢本文，请点击右上角，把文章分享到朋友圈
因公众号更改推送规则，请点“在看”并加“星标”第一时间获取精彩技术分享
·END·
收录于合集 #linux 13个 上一篇Linux命令之精确控制文件访问权限setfacl 阅读 176