jq命令用法總結

扣釘日記發表於2022-05-29

原創：扣釘日記（微信公眾號ID：codelogs），歡迎分享，轉載請保留出處。

簡介

如果說要給Linux文字三劍客(grep、sed、awk)新增一員的話，我覺得應該是jq命令，因為jq命令是用來處理json資料的工具，而現如今json幾乎無所不在！

網上的jq命令分享文章也不少，但大多介紹得非常淺，jq的強大之處完全沒有介紹出來，所以就有了這篇文章，安利一下jq這個命令。

基本用法

格式化

# jq預設的格式化輸出
$ echo -n '{"id":1, "name":"zhangsan", "score":[75, 85, 90]}'|jq .
{
  "id": 1,
  "name": "zhangsan",
  "score": [
    75,
    85,
    90
  ]
}

# -c選項則是壓縮到1行輸出
$ jq -c . <<eof
{
  "id": 1,
  "name": "zhangsan",
  "score": [
    75,
    85,
    90
  ]
}
eof
{"id":1,"name":"zhangsan","score":[75,85,90]}

屬性提取

# 獲取id欄位
$ echo -n '{"id":1, "name":"zhangsan", "score":[75, 85, 90]}'|jq '.id'
1
# 獲取name欄位
$ echo -n '{"id":1, "name":"zhangsan", "score":[75, 85, 90]}'|jq '.name'
"zhangsan"

# 獲取name欄位，-r 解開字串引號
$ echo -n '{"id":1, "name":"zhangsan", "score":[75, 85, 90]}'|jq -r '.name'
zhangsan

# 多層屬性值獲取
$ echo -n '{"id":1, "name":"zhangsan", "attr":{"height":1.78,"weight":"60kg"}}'|jq '.attr.height'
1.78

# 獲取陣列中的值
$ echo -n '{"id":1, "name":"zhangsan", "score":[75, 85, 90]}'|jq -r '.score[0]'
75

$ echo -n '[75, 85, 90]'|jq -r '.[0]'
75

# 陣列擷取
$ echo -n '[75, 85, 90]'|jq -r '.[1:3]'
[
  85,
  90
]

# []展開陣列
$ echo -n '[75, 85, 90]'|jq '.[]'
75
85
90

# ..展開所有結構
$ echo -n '{"id":1, "name":"zhangsan", "score":[75, 85, 90]}'|jq -c '..'
{"id":1,"name":"zhangsan","score":[75,85,90]}
1
"zhangsan"
[75,85,90]
75
85
90

# 從非物件型別中提取欄位，會報錯
$ echo -n '{"id":1, "name":"zhangsan", "attr":{"height":1.78,"weight":"60kg"}}'|jq '.name.alias'
jq: error (at <stdin>:0): Cannot index string with string "alias"

# 使用?號可以避免這種報錯
$ echo -n '{"id":1, "name":"zhangsan", "attr":{"height":1.78,"weight":"60kg"}}'|jq '.name.alias?'

# //符號用於，當前面的表示式取不到值時，執行後面的表示式
$ echo -n '{"id":1, "name":"zhangsan", "attr":{"height":1.78,"weight":"60kg"}}'|jq '.alias//.name'
"zhangsan"

管道、逗號與括號

# 管道可以將值從前一個命令傳送到後一個命令
$ echo -n '{"id":1, "name":"zhangsan", "attr":{"height":1.78,"weight":"60kg"}}'|jq '.attr|.height'
1.78

# jq中做一些基礎運算也是可以的
$ echo -n '{"id":1, "name":"zhangsan", "attr":{"height":1.78,"weight":"60kg"}}'|jq '.attr|.height*100|tostring + "cm"'
"178cm"

# 逗號使得可以執行多個jq表示式，使得一個輸入可計算出多個輸出結果
$ echo 1 | jq '., ., .'
1
1
1

# 括號用於提升表示式的優先順序，如下：逗號優先順序低於算術運算
$ $ echo '1'|jq '.+1, .*2'
2
2

$ echo '1'|jq '(.+1, .)*2'
4
2

# 管道優先順序低於逗號
$ echo '1'|jq '., .|tostring'
"1"
"1"

$ echo '1'|jq '., (.|tostring)'
1
"1"

理解jq執行過程

表面上jq是用來處理json資料的，但實際上jq能處理的是任何json基礎元素所形成的流，如integer、string、bool、null、object、array等，jq執行過程大致如下：

jq從流中獲取一個json元素
jq執行表示式，表示式生成新的json元素
jq將新的json元素列印輸出

可以看看這些示例，如下：

# 這裡jq實際上將1 2 3 4當作4個integer元素，每找到一個元素就執行+1操作
# jq實際上是流式處理的，1 2 3 4可以看成流中的4個元素
$ echo '1 2 3 4'|jq '. + 1'
2
3
4
5

# 流中的元素不需要是同種型別，只要是完整的json元素即可
$ jq '"<" + tostring + ">"' <<eof
1
"zhangsan"
true
{"id":1}
[75, 80, 85]
eof

"<1>"
"<zhangsan>"
"<true>"
"<{\"id\":1}>"
"<[75,80,85]>"

# -R選項可用於將讀取到的json元素，都當作字串對待
$ seq 4|jq -R '.'
"1"
"2"
"3"
"4"

# -s選項將從流中讀取到的所有json元素，變成一個json陣列元素  
# 這裡理解為jq從流中只取到了1個json元素，這個json元素的型別是陣列
$ seq 4|jq -s .
[
  1,
  2,
  3,
  4
]

基礎運算

jq支援 + - * / % 運算，對於+號，如果是字串型別，則是做字串拼接，如下：

# 做加減乘除運算
$ echo 1|jq '.+1, .-1, .*2, ./2, .%2'
2
0
2
0.5
1

# 賦值運算
$ echo -n '{"id":1,"name":"zhangsan","age":"17","score":"75"}'|jq '.id=2' -c
{"id":2,"name":"zhangsan","age":"17","score":"75"}

資料構造

jq可以很方便的將其它資料，轉化為json物件或陣列，如下：

# 使用[]構造陣列元素，-n告訴jq沒有輸入資料，直接執行表示式並生成輸出資料
$ jq -n '[1,2,3,4]' -c
[1,2,3,4]

$ cat data.txt
id  name      age  score
1   zhangsan  17   75
2   lisi      16   80
3   wangwu    18   85
4   zhaoliu   18   90

# 每行分割成陣列，[]構造新的陣列輸出
$ tail -n+2 data.txt|jq -R '[splits("\\s+")]' -c
["1","zhangsan","17","75"]
["2","lisi","16","80"]
["3","wangwu","18","85"]
["4","zhaoliu","18","90"]

$ jq -n '{id:1, name:"zhangsan"}' -c
{"id":1,"name":"zhangsan"}

# 每行轉換為物件，{}構造新的物件格式輸出
$ tail -n+2 data.txt|jq -R '[splits("\\s+")] | {id:.[0]|tonumber, name:.[1], age:.[2], score:.[3]}' -c
{"id":1,"name":"zhangsan","age":"17","score":"75"}
{"id":2,"name":"lisi","age":"16","score":"80"}
{"id":3,"name":"wangwu","age":"18","score":"85"}
{"id":4,"name":"zhaoliu","age":"18","score":"90"}

# \()字串佔位變數替換
$ cat data.json
{"id":1,"name":"zhangsan","age":"17","score":"75"}
{"id":2,"name":"lisi","age":"16","score":"80"}
{"id":3,"name":"wangwu","age":"18","score":"85"}
{"id":4,"name":"zhaoliu","age":"18","score":"90"}

$ cat data.json |jq '"id:\(.id),name:\(.name),age:\(.age),score:\(.score)"' -r
id:1,name:zhangsan,age:17,score:75
id:2,name:lisi,age:16,score:80
id:3,name:wangwu,age:18,score:85
id:4,name:zhaoliu,age:18,score:90

基礎函式

# has函式，檢測物件是否包含key
$ echo -n '{"id":1,"name":"zhangsan","age":"17","score":"75"}'|jq 'has("id")'
true

# del函式，刪除某個屬性
$ echo -n '{"id":1,"name":"zhangsan","age":"17","score":"75"}'|jq 'del(.id)' -c
{"name":"zhangsan","age":"17","score":"75"}

# map函式，對陣列中每個元素執行表示式計算，計算結果組織成新陣列
$ seq 4|jq -s 'map(. * 2)' -c
[2,4,6,8]

# 上面map函式寫法，其實等價於這個寫法
$ seq 4|jq -s '[.[]|.*2]' -c
[2,4,6,8]

# keys函式，列出物件屬性
$ echo -n '{"id":1,"name":"zhangsan","age":"17","score":"75"}'|jq 'keys' -c
["age","id","name","score"]

# to_entries函式，列出物件鍵值對
$ echo -n '{"id":1,"name":"zhangsan","age":"17","score":"75"}'|jq 'to_entries' -c
[{"key":"id","value":1},{"key":"name","value":"zhangsan"},{"key":"age","value":"17"},{"key":"score","value":"75"}]

# length函式，計算陣列或字串長度
$ jq -n '[1,2,3,4]|length'
4

# add函式，計算陣列中數值之和
$ seq 4|jq -s 'add'
10

# tostring與tonumber，型別轉換
$ seq 4|jq 'tostring|tonumber'
1
2
3
4

# type函式，獲取元素型別
$ jq 'type' <<eof
1
"zhangsan"
true
null
{"id":1}
[75, 80, 85]
eof

"number"
"string"
"boolean"
"null"
"object"
"array"

過濾、排序、分組函式

$ cat data.json
{"id":1,"name":"zhangsan","sex": 0, "age":"17","score":"75"}
{"id":2,"name":"lisi","sex": 1, "age":"16","score":"80"}
{"id":3,"name":"wangwu","sex": 0, "age":"18","score":"85"}
{"id":4,"name":"zhaoliu","sex": 0, "age":"18","score":"90"}

# select函式用於過濾，類似SQL中的where
$ cat data.json |jq 'select( (.id>1) and (.age|IN("16","17","18")) and (.name != "lisi") or (has("attr")|not) and (.score|tonumber >= 90) )' -c
{"id":3,"name":"wangwu","sex":0,"age":"18","score":"85"}
{"id":4,"name":"zhaoliu","sex":0,"age":"18","score":"90"}

# 有一些簡化的過濾函式，如arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars
# 它們根據型別過濾，如objects過濾出物件，values過濾出非null值等
$ jq -c 'objects' <<eof
1
"zhangsan"
true
null
{"id":1}
[75, 80, 85]
eof

{"id":1}

$ jq -c 'values' <<eof
1
"zhangsan"
true
null
{"id":1}
[75, 80, 85]
eof

1
"zhangsan"
true
{"id":1}
[75,80,85]

# 選擇出id與name欄位，類似SQL中的select id,name
$ cat data.json|jq -s 'map({id,name})[]' -c
{"id":1,"name":"zhangsan"}
{"id":2,"name":"lisi"}
{"id":3,"name":"wangwu"}
{"id":4,"name":"zhaoliu"}

# 提取前2行，類似SQL中的limit 2
$ cat data.json|jq -s 'limit(2; map({id,name})[])' -c
{"id":1,"name":"zhangsan"}
{"id":2,"name":"lisi"}

# 按照age、id排序，類似SQL中的order by age,id
$ cat data.json|jq -s 'sort_by((.age|tonumber), .id)[]' -c
{"id":2,"name":"lisi","sex":1,"age":"16","score":"80"}
{"id":1,"name":"zhangsan","sex":0,"age":"17","score":"75"}
{"id":3,"name":"wangwu","sex":0,"age":"18","score":"85"}
{"id":4,"name":"zhaoliu","sex":0,"age":"18","score":"90"}


# 根據sex與age分組，並每組聚合計算count(*)、avg(score)、max(id)
$ cat data.json |jq -s 'group_by(.sex, .age)[]' -c
[{"id":1,"name":"zhangsan","sex":0,"age":"17","score":"75"}]
[{"id":3,"name":"wangwu","sex":0,"age":"18","score":"85"},{"id":4,"name":"zhaoliu","sex":0,"age":"18","score":"90"}]
[{"id":2,"name":"lisi","sex":1,"age":"16","score":"80"}]

$ cat data.json |jq -s 'group_by(.sex, .age)[]|{sex:.[0].sex, age:.[0].age, count:length, avg_score:map(.score|tonumber)|(add/length), scores:map(.score)|join(","), max_id:map(.id)|max }' -c                 
{"sex":0,"age":"17","count":1,"avg_score":75,"scores":"75","max_id":1}
{"sex":0,"age":"18","count":2,"avg_score":87.5,"scores":"85,90","max_id":4}
{"sex":1,"age":"16","count":1,"avg_score":80,"scores":"80","max_id":2}

字串操作函式

# contains函式，判斷是否包含，實際也可用於判斷陣列是否包含某個元素
$ echo hello | jq -R 'contains("he")'
true

# 判斷是否以he開頭
$ echo hello | jq -R 'startswith("he")'
true

# 判斷是否以llo結尾
$ echo hello | jq -R 'endswith("llo")'
true

# 去掉起始空格
$ echo ' hello '|jq -R 'ltrimstr(" ")|rtrimstr(" ")'
"hello"

# 大小寫轉換
$ echo hello|jq -R 'ascii_upcase'
"HELLO"

$ echo HELLO|jq -R 'ascii_downcase'
"hello"

# 字串陣列，通過逗號拼接成一個字串
$ seq 4|jq -s 'map(tostring)|join(",")'
"1,2,3,4"

# json字串轉換為json物件
$ echo -n '{"id":1,"name":"zhangsan","age":"17","attr":"{\"weight\":56,\"height\":178}"}'|jq '.attr = (.attr|fromjson)' -c
{"id":1,"name":"zhangsan","age":"17","attr":{"weight":56,"height":178}}

# json物件轉換為json字串
$ echo -n '{"id":1,"name":"zhangsan","age":"17","attr":{"weight":56,"height":178}}'|jq '.attr = (.attr|tojson)'
{
  "id": 1,
  "name": "zhangsan",
  "age": "17",
  "attr": "{\"weight\":56,\"height\":178}"
}

$ cat data.txt
id:1,name:zhangsan,age:17,score:75
id:2,name:lisi,age:16,score:80
id:3,name:wangwu,age:18,score:85
id:4,name:zhaoliu,age:18,score:90

# 正規表示式過濾，jq使用的是PCRE
$ cat data.txt|jq -R 'select(test("id:\\d+,name:\\w+,age:\\d+,score:8\\d+"))' -r
id:2,name:lisi,age:16,score:80
id:3,name:wangwu,age:18,score:85

# 正則拆分字串
$ cat data.txt|jq -R '[splits(",")]' -cr
["id:1","name:zhangsan","age:17","score:75"]
["id:2","name:lisi","age:16","score:80"]
["id:3","name:wangwu","age:18","score:85"]
["id:4","name:zhaoliu","age:18","score:90"]

# 正則替換字串
$ cat data.txt |jq -R 'gsub("name"; "nick")' -r
id:1,nick:zhangsan,age:17,score:75
id:2,nick:lisi,age:16,score:80
id:3,nick:wangwu,age:18,score:85
id:4,nick:zhaoliu,age:18,score:90

# 正規表示式捕獲資料
$ cat data.txt|jq -R 'match("id:(?<id>\\d+),name:(?<name>\\w+),age:\\d+,score:8\\d+")' -cr
{"offset":0,"length":30,"string":"id:2,name:lisi,age:16,score:80","captures":[{"offset":3,"length":1,"string":"2","name":"id"},{"offset":10,"length":4,"string":"lisi","name":"name"}]}
{"offset":0,"length":32,"string":"id:3,name:wangwu,age:18,score:85","captures":[{"offset":3,"length":1,"string":"3","name":"id"},{"offset":10,"length":6,"string":"wangwu","name":"name"}]}

# capture命名捕獲，生成key是捕獲組名稱，value是捕獲值的物件
$ cat data.txt|jq -R 'capture("id:(?<id>\\d+),name:(?<name>\\w+),age:\\d+,score:8\\d+")' -rc
{"id":"2","name":"lisi"}
{"id":"3","name":"wangwu"}

# 正則掃描輸入字串
$ cat data.txt|jq -R '[scan("\\w+:\\w+")]' -rc
["id:1","name:zhangsan","age:17","score:75"]
["id:2","name:lisi","age:16","score:80"]
["id:3","name:wangwu","age:18","score:85"]
["id:4","name:zhaoliu","age:18","score:90"]

日期函式

# 當前時間綴
$ jq -n 'now'
1653820640.939947

# 將時間綴轉換為0時區的分解時間(broken down time)，形式為 年 月 日 時 分 秒 dayOfWeek dayOfYear
$ jq -n 'now|gmtime' -c
[2022,4,29,10,45,5.466768980026245,0,148]

# 將時間綴轉換為本地時區的分解時間(broken down time)
$ jq -n 'now|localtime' -c
[2022,4,29,18,46,5.386353015899658,0,148]

# 分解時間轉換為時間串
$ jq -n 'now|localtime|strftime("%Y-%m-%dT%H:%M:%S")' -c
"2022-05-29T18:50:33"

# 與上面等效
$ jq -n 'now|strflocaltime("%Y-%m-%dT%H:%M:%SZ")'
"2022-05-29T19:00:40Z"

# 時間串解析為分解時間
$ date +%FT%T|jq -R 'strptime("%Y-%m-%dT%H:%M:%S")' -c
[2022,4,29,18,51,27,0,148]

# 分解時間轉換為時間綴
$ date +%FT%T|jq -R 'strptime("%Y-%m-%dT%H:%M:%S")|mktime'
1653850310

高階用法

實際上jq是一門指令碼語言，它也支援變數、分支結構、迴圈結構與自定義函式，如下：

$ cat data.json
{"id":1,"name":"zhangsan","sex": 0, "age":"17","score":"75"}
{"id":2,"name":"lisi","sex": 1, "age":"16","score":"80"}
{"id":3,"name":"wangwu","sex": 0, "age":"18","score":"85"}
{"id":4,"name":"zhaoliu","sex": 0, "age":"18","score":"90"}

# 單變數定義
$ cat data.json| jq '.id as $id|$id'
1
2
3
4

# 物件展開式變數定義
$ cat data.json |jq '. as {id:$id,name:$name}|"id:\($id),name:\($name)"'
"id:1,name:zhangsan"
"id:2,name:lisi"
"id:3,name:wangwu"
"id:4,name:zhaoliu"

$ cat data.json
["1","zhangsan","17","75"]
["2","lisi","16","80"]
["3","wangwu","18","85"]
["4","zhaoliu","18","90"]

# 陣列展開式變數定義
$ cat data.json|jq '. as [$id,$name]|"id:\($id),name:\($name)"'
"id:1,name:zhangsan"
"id:2,name:lisi"
"id:3,name:wangwu"
"id:4,name:zhaoliu"

# 分支結構
$ cat data.json|jq '. as [$id,$name]|if ($id>"1") then "id:\($id),name:\($name)" else empty end'
"id:2,name:lisi"
"id:3,name:wangwu"
"id:4,name:zhaoliu"

# 迴圈結構，第一個表示式條件滿足時，執行只每二個表示式
# 迴圈結構除了while，還有until、recurse等
$ echo 1|jq 'while(.<100; .*2)'
1
2
4
8
16
32
64

# 自定義計算3次方的函式
$ echo 2|jq 'def cube: .*.*. ; cube'
8

由於這些高階特性並不常用，這裡僅給出了一些簡單示例，詳細使用可以man jq檢視。

輔助shell程式設計

熟悉shell指令碼程式設計的同學都知道，shell本身是沒有提供Map、List這種資料結構的，這導致使用shell實現某些功能時，變得很棘手。

但jq本身是處理json的，而json中的物件就可等同於Map，json中的陣列就可等同於List，如下：

list='[]';
#List新增元素
list=$(echo "$list"|jq '. + [ $val ]' --arg val java);
list=$(echo "$list"|jq '. + [ $val ]' --arg val shell);
#獲取List大小
echo "$list"|jq '.|length'
#獲取List第1個元素
echo "$list"|jq '.[0]' -r
# List是否包含java字串
echo "$list"|jq 'any(.=="java")'
#刪除List第1個元素
list=$(echo "$list"|jq 'del(.[0])');
# List合併
list=$(echo "$list"|jq '. + $val' --argjson val '["shell","python"]');
# List擷取
echo "$list"|jq '.[1:3]'
# List遍歷
for o in $(echo "$list" | jq -r '.[]');do 
    echo "$o"; 
done

map='{}';
#Map新增元素
map=$(echo "$map"|jq '.id=$val' --argjson val 1)
map=$(echo "$map"|jq '.courses=$val' --argjson val "$list")
#獲取Map大小
echo "$map"|jq '.|length'
#獲取Map指定key的值
echo "$map"|jq '.id' -r
#判斷Map指定key是否存在
echo "$map" | jq 'has("id")'
#刪除Map指定key
map=$(echo "$map"|jq 'del(.id)')
# Map合併
map=$(echo "$map"|jq '. + $val' --argjson val '{"code":"ID001","name":"hello"}')
# Map的KeySet遍歷
for key in $(echo "$map" | jq -r 'keys[]'); do 
    value=$(jq '.[$a]' --arg a "$key" -r <<<"$map"); 
    printf "%s:%s\n" "$key" "$value"; 
done
# Map的entrySet遍歷
while read -r line; do 
    key=$(jq '.key' -r <<<"$line"); 
    value=$(jq '.value' -r <<<"$line"); 
    printf "%s:%s\n" "$key" "$value"; 
done <<<$(echo "$map" | jq 'to_entries[]' -c)

總結

可以發現，jq已經實現了json資料處理與分析的方方面面，我個人最近在工作中，也多次使用jq來分析呼叫日誌等，用起來確實非常方便。

如果你現在還沒完全學會jq的用法，沒關係，建議先收藏起來，後面一定會用得到的！

往期內容

密碼學入門
 q命令-用SQL分析文字檔案
 神祕的backlog引數與TCP連線佇列
 mysql的timestamp會存在時區問題？
真正理解可重複讀事務隔離級別
 字元編碼解惑

sed命令用法總結
2016-10-10
linux命令下jq的用法簡介
2022-06-19
Linux
jQ基礎篇–jQuery選擇器總結
2015-08-31
jQuery
pandas用法總結
2020-04-05
layui用法總結
2019-11-21
UI
less用法總結
2019-11-22
Promise用法總結
2018-09-14
Promise
sed用法總結
2013-12-02
Autotrace 用法總結
2012-05-10
linux下find（檔案查詢）命令的用法總結
2017-05-12
Linux
Quartz：基本用法總結
2020-08-21
quartz
axios用法總結
2019-11-21
iOS
Const 用法總結
2019-05-12
xPath 用法總結整理
2019-01-21
STL:set用法總結
2018-10-30
STL:map用法總結
2018-10-30
STL:vector用法總結
2018-10-26
STL:list用法總結
2018-10-26
【Git】stash 用法總結
2017-04-11
Git
iframe元素用法總結
2013-12-10
adb 用法總結
2013-08-02
uniDAC用法總結
2013-06-07
type的用法總結
2014-08-07
oracle hints用法總結
2010-09-14
Oracle
sqlldr的用法總結
2009-12-11
SQL
React屬性用法總結
2018-09-14
React
【Less】Less基本用法總結
2020-12-30
git reset rebase 用法總結
2018-02-08
Git
jquery validate用法總結
2017-08-27
jQuery
Java習慣用法總結
2015-04-07
Java
Htaccess檔案用法總結
2012-11-12
ORACLE 中ROWNUM用法總結!
2007-11-07
Oracle
ORACLE 中ROWNUM用法總結
2009-10-18
Oracle
ORACLE 繫結變數用法總結
2014-11-12
Oracle變數
tkprof命令列工具用法小結
2010-07-12
命令列
NPM 命令總結
2020-02-12
NPM
docker命令總結
2019-12-31
Docker
VIM 命令總結
2019-08-17