那阵子在moto做项目，利用间隙看完了一本o'reilly公司的sed&awk介绍，作了一些笔记。巧的是所学的一些东西在项目后期居然用到非常多，呵呵
1. 关于过滤字符串问题
比如在一个文本中搜索abcd这四个字符，使用grep 结果出来abcdef 、zabcd 、abcd 、 hiabcd.... 等等，其实只想搜索abcd
A: 使用精确匹配 $grep "
比如在一个文本中搜索abcd这四个字符，使用grep 结果出来abcdef 、zabcd 、abcd 、 hiabcd.... 等等，其实只想搜索abcd
A: 使用精确匹配 $grep "
2. sed中查找模式匹配：
. 单字符通配符
[0-9] 匹配0,1,2,3...8,9
[a-z] 匹配a,b,c...z
[^0-9] 匹配不含0,1,2...9的所有其他字符
".*" 匹配""内任何字符串
^ 行开始标志
$ 行结束标志
. 单字符通配符
[0-9] 匹配0,1,2,3...8,9
[a-z] 匹配a,b,c...z
[^0-9] 匹配不含0,1,2...9的所有其他字符
".*" 匹配""内任何字符串
^ 行开始标志
$ 行结束标志
3. sed中用bbb替换同一行中包括字符串and的字符串aaa，而不是每一行中的字符串aaa
A: $sed -e '/and/s/aaa/bbb/' filename
A: $sed -e '/and/s/aaa/bbb/' filename
4. sed删除文件filename中的所有行
A: $sed d filename
A: $sed d filename
5. sed删除文件filename中的2至5行
A: $sed 2,5d filename
A: $sed 2,5d filename
6. sed删除文件filename中包含字符串abc的所有行
A: $sed /abc/d filename
A: $sed /abc/d filename
7. A regular expression can use "n" to match an embeded newline
8. The regular expression can be delimited by any character except a blank or a newline
EX: s!/usr/mail!/usr2/mail!
Note that the delimiter appears three times and is required after the replacement "/usr2/mail"
EX: s!/usr/mail!/usr2/mail!
Note that the delimiter appears three times and is required after the replacement "/usr2/mail"
9. SED Insert: [line-address]i
sed '/
4700 cross court
French Lick, IN' filename
Desc: inserts two lines of text at a line matching "
10. SED Append: [line-address]a
sed '$a
Append a new line ' filename
DESC: append a new line in the end of filename
11. SED Change: [line-address]c
sed '/100/c
one hundred' filename
DESC: change the whole lines containing '100' into 'one hundred'
sed '/
4700 cross court
French Lick, IN' filename
Desc: inserts two lines of text at a line matching "
10. SED Append: [line-address]a
sed '$a
Append a new line ' filename
DESC: append a new line in the end of filename
11. SED Change: [line-address]c
sed '/100/c
one hundred' filename
DESC: change the whole lines containing '100' into 'one hundred'
12. 用SED显示TAB等特殊字符: l
sed -n -e "l" test1
NOTE:显示test1文件的内容，包括TAB等特殊字符。-n过滤重复行
sed -n -e "l" test1
NOTE:显示test1文件的内容，包括TAB等特殊字符。-n过滤重复行
13. SED n:下一行/next p:显示/print
$ sed -n -e '/James/{
> n
> p
> }' filename
DESC: 显示filename中含有James字符串的下一行
$ sed -n -e '/James/{
> n
> p
> }' filename
DESC: 显示filename中含有James字符串的下一行
14. SED q: quit
$sed '100q' filename
DESC: print the first 100 lines from filename
15. SED y: transform
/.*/y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/
DESC:replace all the characters into capital
16. convert file 'filename' from 1 to 2
2 1
11 22
22 11
111 222
222 111
$ sed '/1/{
> h
> d
> }
> /2/{
> G
> }' filename
h/H: Hold, copy or append contents of pattern space to hold space
g/G: Get, copy or append contents of hold space to pattern space
x: Exchange, swap contents of hold space and pattern space
17. AWK中的特殊字符
a alert character, usually ASCII BEL character
b backspace
f formfeed
n newline
r carriage return
t horizontal tab
v vertical tab
c any literal character c
$sed '100q' filename
DESC: print the first 100 lines from filename
15. SED y: transform
/.*/y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/
DESC:replace all the characters into capital
16. convert file 'filename' from 1 to 2
2 1
11 22
22 11
111 222
222 111
$ sed '/1/{
> h
> d
> }
> /2/{
> G
> }' filename
h/H: Hold, copy or append contents of pattern space to hold space
g/G: Get, copy or append contents of hold space to pattern space
x: Exchange, swap contents of hold space and pattern space
17. AWK中的特殊字符
a alert character, usually ASCII BEL character
b backspace
f formfeed
n newline
r carriage return
t horizontal tab
v vertical tab
c any literal character c
18. AWD系统变量
FS field separator
NR the number of current input record
FILENAME the name of current input file
RS the record separator, as a newline
NF the number of fields for the current record
FS field separator
NR the number of current input record
FILENAME the name of current input file
RS the record separator, as a newline
NF the number of fields for the current record
EXAMPLE awkcmd 列出各纪录平均值，并标出行号，最后显示文件名和记录数:
#list the avg scores and row numbers
BEGIN { FS=" " }
{ print NR,".",$1,",",($2+$3+$4)/3 }
END { print
print "File " FILENAME " has " NR " rows " }
$awk -f awkcmd sourcefile
#list the avg scores and row numbers
BEGIN { FS=" " }
{ print NR,".",$1,",",($2+$3+$4)/3 }
END { print
print "File " FILENAME " has " NR " rows " }
$awk -f awkcmd sourcefile
19. #将多行的块纪录转成单行纪录
BEGIN { FS="n"; RS="" }
{ print $1, $NF }
20. 显示当前目录下各文件名和大小的例子
ls -lR|awk '
BEGIN {print " size"," ", "filename"}
NF==9 && /^-/ { #9列并以-开头表示文件
filesize += $5
++filenum
print " file ", $5," ",$9
}
NF==9 && /^d/ { #9列并以d开头表示子目录
print "", "t", $9
}
$1 ~ /^..*:$/ { # ~: Match Operator
#ls -lR中子目录显示为 ./subdir: 以"."开头"$"结尾的行判断为子目录的行
print "t" $0
}
END { print "Total: ", filesize, " bytes (", filenum, " files)"} '
21. AWK传递参数和检索数据
$ cat awk_par
awk '$2==search' search=$1 srcfile
运行:
$ awk_par par_val
DESC: 参数par_val传递给变量search，显示srcfile中第二列为par_val的记录行
22. AWK中将shell参数传递给awk
search=$1
awk '$2 ~ /'"${search}"'/' srcfile
DESC: 将shell的第一个参数传递给变量search，同srcfile中第二列比较
如果改成 awk '$2 ~ /'"${search:-.*}"'/' srcfile，则如果shell运行时无参数$1,就显示所有记录
BEGIN { FS="n"; RS="" }
{ print $1, $NF }
20. 显示当前目录下各文件名和大小的例子
ls -lR|awk '
BEGIN {print " size"," ", "filename"}
NF==9 && /^-/ { #9列并以-开头表示文件
filesize += $5
++filenum
print " file ", $5," ",$9
}
NF==9 && /^d/ { #9列并以d开头表示子目录
print "
}
$1 ~ /^..*:$/ { # ~: Match Operator
#ls -lR中子目录显示为 ./subdir: 以"."开头"$"结尾的行判断为子目录的行
print "t" $0
}
END { print "Total: ", filesize, " bytes (", filenum, " files)"} '
21. AWK传递参数和检索数据
$ cat awk_par
awk '$2==search' search=$1 srcfile
运行:
$ awk_par par_val
DESC: 参数par_val传递给变量search，显示srcfile中第二列为par_val的记录行
22. AWK中将shell参数传递给awk
search=$1
awk '$2 ~ /'"${search}"'/' srcfile
DESC: 将shell的第一个参数传递给变量search，同srcfile中第二列比较
如果改成 awk '$2 ~ /'"${search:-.*}"'/' srcfile，则如果shell运行时无参数$1,就显示所有记录