在前文的基礎上，增加了改進。
主要是細節格式的調整.
處理100M資料用時7s

http://blog.itpub.net/29254281/viewspace-2156713/

time awk '
function ltrim(s) { sub(/^[ \t\r\n]+/, "", s); return s }
function rtrim(s) { sub(/[ \t\r\n]+$/, "", s); return s }
function trim(s) { return rtrim(ltrim(s)); }
/^Query Id:/{split($0,rowdata,":");}
/^SQL:/{s=trim($0);next;}
/^User:/{print "19700101 000000000 INFO [Query "trim(rowdata[2])" "s;s="";print $0;}
/^==========================/{delete rowdata;}
{
if(rowdata[2]!="")
if(s!="") s=(s" "trim($0))
else
print "19700101 000000000 INFO [Query "trim(rowdata[2])" "$0;
else gsub("[,:-]","",$1);gsub("[,:-]","",$2);$5=substr($5,0,36);print ($0);
}

' kylinbig.log | grep "\\[Query" | nl -n ln | tr "\t" " " | tr -s ' ' ' ' | sort -k6,6 -k1n,1 | cut -d ' ' -f 2,3,4,6- | sed '/Using project:/{x;p;x;}' > t1.txt

使用AWK規整KyLin日誌改進

相關文章