Pytorch 自動微分

• Tensor.requires_grad = True 記錄對Tensor的所有操作，後序.backward() 自動計算所有梯度到 .grad 屬性

import torch
x = torch.ones(2,2, requires_grad=True) # 預設是False
print(x)

tensor([[1., 1.],
        [1., 1.]], requires_grad=True)

• 停止記錄呼叫.detach()

x.detach_()
print(x.requires_grad) # False

• .grad_fn 儲存了建立張量的 Function 的引用

y = x + 2
print(y)
print(y.grad_fn)

tensor([[3., 3.],
        [3., 3.]], grad_fn=<AddBackward0>)
<AddBackward0 object at 0x0000015716529D68>

z = y*y*3
out = z.mean()
print(z, out)

tensor([[27., 27.],
        [27., 27.]], grad_fn=<MulBackward0>) 

tensor(27., grad_fn=<MeanBackward0>)

# requires_grad 預設為 False
a = torch.randn(2, 2)
a = ((a*3)/(a-1))
print(a.requires_grad)  # False
b = (a*a).sum()
print(b.grad_fn)  # None

a.requires_grad_(True)  # 設定為 True
print(a.requires_grad)  # True
b = (a*a).sum()
print(b.grad_fn)
# <SumBackward0 object at 0x0000015717DC69E8>

• backward() 後向傳播

z = y*y*3
y = x+2
計算 d(out)/dx

$out=\frac{1}{4}(\sum 3(x_i+2{)}^2)\to \frac{d_{out}}{d{x}_i}=\frac{3}{2}(x_i+2)$
$x_i=1,d_{out}/d{x}_i=4.5$

out.backward()
print(y.grad) # None, 為什麼？是 None
print(x.grad)
tensor([[4.5000, 4.5000],
        [4.5000, 4.5000]])

$$J=\left(\begin{array}{ccc}\frac{\partial y_1}{\partial x_1}& \cdots & \frac{\partial y_m}{\partial x_1}\\ ⋮& \ddots & ⋮\\ \frac{\partial y_1}{\partial x_n}& \cdots & \frac{\partial y_m}{\partial x_n}\end{array}\right)$$

• 當又使用了一個函式 $l=g(y)$，v 是 $l$ 對 $y$ 的導數，鏈式求導相乘，得到 $l$ 對 $x$ 的導數
  $$J\cdot v=\left(\begin{array}{ccc}\frac{\partial y_1}{\partial x_1}& \cdots & \frac{\partial y_m}{\partial x_1}\\ ⋮& \ddots & ⋮\\ \frac{\partial y_1}{\partial x_n}& \cdots & \frac{\partial y_m}{\partial x_n}\end{array}\right)\left(\begin{array}{c}\frac{\partial l}{\partial y_1}\\ ⋮\\ \frac{\partial l}{\partial y_m}\end{array}\right)=\left(\begin{array}{c}\frac{\partial l}{\partial x_1}\\ ⋮\\ \frac{\partial l}{\partial x_n}\end{array}\right)$$

上面程式碼改為：

v = torch.tensor(2, dtype=torch.float)
out.backward(v)
print(x.grad)

# 梯度乘以了 2
tensor([[9., 9.],
        [9., 9.]])

• 評估階段可以使用 with torch.no_grad(): 不需要梯度計算和更新

print(x.requires_grad) # True
print((x ** 2).requires_grad) # True

# 取消梯度記錄
with torch.no_grad():
    print((x ** 2).requires_grad) # False