【Linux】jq 命令介紹和使用

Xander發表於2023-03-06
Linux

介紹

jq 命令主要是和json檔案的操作有關的輔助命令，可以實現對於Json資料的切片，過濾，對映和轉換，在功能上毫不遜色於awk、sed、grep等命令。

注意：這裡的jq和我們認知的前端的那個"jq"是兩碼事，不要混入前端框架的概念。

jq是用C編寫，沒有執行時依賴，所以幾乎可以執行在任何系統上。可以跨各種常用的作業系統，雖然官方沒直接提供CenterOs系統的下載方式，但是可以透過新增epel源的方式處理。

作用

可以讓linux命令和shell指令碼在處理json資料時變得得心應手。

官方案例演示

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5'

結果如下：

格式化輸出：

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.'

獲取第一個元素：

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.[0]'
[zxd@localhost ~]$ curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.[0]'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19313  100 19313    0     0   3131      0  0:00:06  0:00:06 --:--:--  4147
{
  "sha": "cff5336ec71b6fee396a95bb0e4bea365e0cd1e8",
  "node_id": "C_kwDOAE3WVdoAKGNmZjUzMzZlYzcxYjZmZWUzOTZhOTViYjBlNGJlYTM2NWUwY2QxZTg",
  "commit": {
    "author": {
      "name": "Mattias Wadman",
      "email": "mattias.wadman@gmail.com",
      "date": "2021-06-09T14:02:22Z"
    },
    "committer": {
      "name": "Nico Williams",
      "email": "nico@cryptonector.com",
      "date": "2022-05-26T21:04:32Z"
    },
    "message": "docs: Document repeat(exp)",
    "tree": {
      "sha": "d67d5542df1f16d1a48e1fb75749f60482cd874b",
      "url": "https://api.github.com/repos/stedolan/jq/git/trees/d67d5542df1f16d1a48e1fb75749f60482cd874b"
    },
    "url": "https://api.github.com/repos/stedolan/jq/git/commits/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8",
    "comment_count": 0,
    "verification": {
      "verified": false,
      "reason": "unsigned",
      "signature": null,
      "payload": null
    }
  },
  "url": "https://api.github.com/repos/stedolan/jq/commits/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8",
  "html_url": "https://github.com/stedolan/jq/commit/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8",
  "comments_url": "https://api.github.com/repos/stedolan/jq/commits/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8/comments",
  "author": {
    "login": "wader",
    "id": 185566,
    "node_id": "MDQ6VXNlcjE4NTU2Ng==",
    "avatar_url": "https://avatars.githubusercontent.com/u/185566?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/wader",
    "html_url": "https://github.com/wader",
    "followers_url": "https://api.github.com/users/wader/followers",
    "following_url": "https://api.github.com/users/wader/following{/other_user}",
    "gists_url": "https://api.github.com/users/wader/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/wader/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/wader/subscriptions",
    "organizations_url": "https://api.github.com/users/wader/orgs",
    "repos_url": "https://api.github.com/users/wader/repos",
    "events_url": "https://api.github.com/users/wader/events{/privacy}",
    "received_events_url": "https://api.github.com/users/wader/received_events",
    "type": "User",
    "site_admin": false
  },
  "committer": {
    "login": "nicowilliams",
    "id": 604851,
    "node_id": "MDQ6VXNlcjYwNDg1MQ==",
    "avatar_url": "https://avatars.githubusercontent.com/u/604851?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/nicowilliams",
    "html_url": "https://github.com/nicowilliams",
    "followers_url": "https://api.github.com/users/nicowilliams/followers",
    "following_url": "https://api.github.com/users/nicowilliams/following{/other_user}",
    "gists_url": "https://api.github.com/users/nicowilliams/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/nicowilliams/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/nicowilliams/subscriptions",
    "organizations_url": "https://api.github.com/users/nicowilliams/orgs",
    "repos_url": "https://api.github.com/users/nicowilliams/repos",
    "events_url": "https://api.github.com/users/nicowilliams/events{/privacy}",
    "received_events_url": "https://api.github.com/users/nicowilliams/received_events",
    "type": "User",
    "site_admin": false
  },
  "parents": [
    {
      "sha": "f2ad9517c72f6267ae317639ab56bbfd4a8653d4",
      "url": "https://api.github.com/repos/stedolan/jq/commits/f2ad9517c72f6267ae317639ab56bbfd4a8653d4",
      "html_url": "https://github.com/stedolan/jq/commit/f2ad9517c72f6267ae317639ab56bbfd4a8653d4"
    }
  ]
}

下載地址：

github地址：

下面介紹比較常用的Ubuntu和CenterOs的安裝教程。

Ubuntu 安裝

安裝方式非常簡單，直接按照官方教程即可。

jq 1.5 is in the official Debian and Ubuntu repositories. Install using sudo apt-get install jq.
sudo apt-get install jq
ubuntu@VM-8-8-ubuntu:~$ sudo apt-get install jq
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
..
等待安裝完成即可

我們直接輸入jq命令驗證是否安裝成功：

ubuntu@VM-8-8-ubuntu:~$ jq 
jq - commandline JSON processor [version 1.6]

Usage:    jq [options] <jq filter> [file...]
    jq [options] --args <jq filter> [strings...]
    jq [options] --jsonargs <jq filter> [JSON_TEXTS...]

jq is a tool for processing JSON inputs, applying the given filter to
its JSON text inputs and producing the filter's results as JSON on
standard output.

The simplest filter is ., which copies jq's input to its output
unmodified (except for formatting, but note that IEEE754 is used
for number representation internally, with all that that implies).

For more advanced filters see the jq(1) manpage ("man jq")
and/or https://stedolan.github.io/jq

Example:

    $ echo '{"foo": 0}' | jq .
    {
        "foo": 0
    }

For a listing of options, use jq --help.
ubuntu@VM-8-8-ubuntu:~$ yum install epel-release
Command 'yum' not found, did you mean:
  command 'gum' from snap gum (0.12.0)
  command 'yum4' from deb nextgen-yum4 (4.5.2-6)
  command 'num' from deb quickcal (2.4-1)
  command 'zum' from deb perforate (1.2-5.1)
  command 'sum' from deb coreutils (8.32-4.1ubuntu1)
  command 'uum' from deb freewnn-jserver (1.1.1~a021+cvs20130302-7build1)
See 'snap info <snapname>' for additional versions.

CenterOs 安裝

使用Centos7.9 版本進行安裝稍微麻煩一丟丟，查閱官方介紹發現並沒有CenterOs系統的安裝方式，這是因為需要新增相關的源才行：

  1. 安裝EPEL源：
yum install -y epel-release
  1. 安裝完EPEL源後，可以檢視下jq包是否存在：
yum list jq
[zxd@localhost ~]$ yum list jq
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager
Determining fastest mirrors
epel/x86_64/metalink                                                                            | 7.6 kB  00:00:00     
 * base: mirrors.bupt.edu.cn
 * epel: mirrors.bfsu.edu.cn
 * extras: mirrors.bupt.edu.cn
 * updates: mirrors.bupt.edu.cn
epel                                                                                            | 4.7 kB  00:00:00     
(1/3): epel/x86_64/group_gz                                                                     |  99 kB  00:00:00     
(2/3): epel/x86_64/updateinfo                                                                   | 1.0 MB  00:00:01     
(3/3): epel/x86_64/primary_db                                                                   | 7.0 MB  00:00:03     
rabbitmq_erlang                                                                                                  47/47
rabbitmq_rabbitmq-server                                                                                         98/98
Available Packages
jq.x86_64                                                1.6-2.el7                                                 epe
  1. 安裝jq：
yum install jq

安裝完成之後同樣可以直接輸入命令jq驗證一下。

快速上手

我們可以使用tldr命令檢視一些案例用法，tldr是什麼可以看這篇文章：[[【Linux】Linux命令快速學習神器tldr、cheat介紹和使用]]

[zxd@localhost ~]$ tldr jq

  jq

  A command-line JSON processor that uses a domain-specific language.
  More information: https://stedolan.github.io/jq/manual/.

  - Execute a specific expression (print a colored and formatted json):
  - 最簡單用法，查詢JSON 檔案並且格式化
    cat path/to/file.json | jq '.'

  - Execute a specific script:
    cat path/to/file.json | jq --from-file path/to/script.jq

  - Pass specific arguments:
    cat path/to/file.json | jq --arg "name1" "value1" --arg "name2" "value2" ... '. + $ARGS.named'

  - Print specific keys:
    cat path/to/file.json | jq '.key1, .key2, ...'

  - Print specific array items:
    cat path/to/file.json | jq '.[index1], .[index2], ...'

  - Print all array items/object keys:
    cat path/to/file.json | jq '.[]'

  - Add/remove specific keys:
    cat path/to/file.json | jq '. +|- {"key1": "value1", "key2": "value2", ...}'

簡單使用

"." 最簡單用法

表示式"."作用是格式化輸出JSON：

cat test.json | jq "."
[zxd@localhost ~]$ cat test.json | jq "."
[
  {
    "name": "xxxxxx",
    "url": "http://tool.xxxxx.com",
    "address": {
      "city": "廈門",
      "country": "中國"
    },
    "arrayBrowser": [
      {
        "name": "Google",
        "url": "http://www.google.com"
      },
      {
        "name": "Baidu",
        "url": "http://www.baidu.com"
      }
    ]
  },
  {
    "name": "xxxxxx",
    "url": "http://tool.xxxxx.com",
    "address": {
      "city": "大連",
      "country": "中國"
    },
    "arrayBrowser": [
      {
        "name": "360",
        "url": "http://www.so.com"
      },
      {
        "name": "bing",
        "url": "http://www.bing.com"
      }
    ]
  }
]

[index]

輸出列表中的第一個元素，可以使用[index]：

cat test.json | jq ".[0]"
[zxd@localhost ~]$ cat test.json | jq ".[0]"
{
  "name": "xxxxxx",
  "url": "http://tool.chinaz.com",
  "address": {
    "city": "廈門",
    "country": "中國"
  },
  "arrayBrowser": [
    {
      "name": "Google",
      "url": "http://www.google.com"
    },
    {
      "name": "Baidu",
      "url": "http://www.baidu.com"
    }
  ]
}

管道符 |

| 管道符號是非常常用的符號，用於把一個命令的執行結果轉叫給另一個命令，就像是流水線加工一樣：

這裡的示例是訪問第一個元素，進而訪問巢狀的屬性，如.name和.address.city。

cat json.txt | jq '.[0] | {name:.name,city:.address.city}'
[zxd@localhost ~]$ cat test.json | jq '.[0] | {name:.name,city:.address.city}'
{
  "name": "站長工具",
  "city": "廈門"
}

cat test.json | jq '.[0] | {name:.arrayBrowser[1].name,city:.address.city}'
{
  "name": "Baidu",
  "city": "廈門"
}
cat test.json | jq ".[] | {name:.arrayBrowser[1].name,city:.address.city}"
{
  "name": "Baidu",
  "city": "廈門"
}
{
  "name": "bing",
  "city": "大連"
}

輸出作為陣列

把Jq的輸出內容轉為陣列只需要在對應的位置新增[]一對括號即可：

cat test.json | jq "[.[] | {name:.arrayBrowser[1].name,city:.address.city}]"
[zxd@localhost ~]$ cat test.json | jq "[.[] | {name:.arrayBrowser[1].name,city:.address.city}]"
[
  {
    "name": "Baidu",
    "city": "廈門"
  },
  {
    "name": "bing",
    "city": "大連"
  }
]

key 對映名稱修改

在{}中，冒號前面的名字是對映的名稱，你可以任意修改，如下：

cat json.txt | jq "[.[] | {name_001:.arrayBrowser[1].name,city_002:.address.city}]"
[
  {
    "name_001": "Baidu",
    "city_002": "廈門"
  },
  {
    "name_001": "bing",
    "city_002": "大連"
  }
]

參考資料

https://www.jianshu.com/p/6de3cfdbdb0e

擴充套件閱讀

http://www.json.cn/wiki.html

https://stedolan.github.io/jq/tutorial/

相關文章