transformer

默盒發表於2021-02-17
ORM

Transformer Model

性質:

1. Transformer是Seq2Seq類模型.
2. ransformer不是RNN.
3.僅依賴attention和全連線層.
準確率遠高於RNN類.

各種weights:

  1. \(weights \space\space \alpha_{ij} = align(h_i, s_j)\).
  2. Compute\(k_{:i} = W_K h_i\)and\(q_{:j} = W_Q S_j\).
  3. Compute weights\(\alpha_{:j} = Softmax(K^T q_{:j}) \in \mathbb{R}^m\).
  4. Context vector:\(c_j = \sum\limits_{i=1}^{m}{\alpha_{ij}v_{:m}}\).
  • Query:\(q_{:j} = W_Q s_j\)-- 匹配別人.
  • Key:\(k_{:i} = W_K h_i\)-- 等待被匹配.
  • Value:\(V_{:i} = W_V h_i\)-- 待加權平均.
  • \(W_Q, W_K, W_V\)皆為待學習引數.

\(Q-K-V\)的關係其實就是:\(h(P)與s(P)求對於h(P)的 attention\), 三個(P)處都是不同的可學習的W.

Attention Layer

Key:\(k_{:i} = W_K x_i\).

Value:\(v_{:i} = W_V x_i\).

  • Queries are based on decoder's inputs\(x_1^\prime, x_2^\prime, ..., x_t^\prime\).
  • Query:\(q_{:j} = W_Q x_j^\prime\).

符號彙總:

  • Attention layer:\(C = Attn(X, X^\prime)\).
    • Encoder's inputs:\(X = [x_1, x_2, ..., x_m]\).
    • Decoder's inputs:\(X^\prime = [x_1^\prime, x_2^\prime, ..., x_t^\prime]\).
    • parameters:\(W_Q, W_K, W_V\).

  • Self-attention layer:\(C = Attn(X, X)\).
    • RNN's inputs\(X = [x_1, x_2, ..., x_m]\).
    • Parameters:\(W_Q, W_K, W_V\).

Summary:

  • Attention 最初用於Seq2Seq的RNN模型.
  • self-attention: 可用於所有RNN模型而不僅是Seq2Seq模型.
  • Attention 可以不依賴於RNN使用.

Transformer 架構:

Single-head self-attention

Multi-head self-attention:

  • l 個不共享權重的single-head self-attentions.
  • 將所有single-head self-attentions的結果concat起來
    • 假設single-head self-attention的輸出為dxm的矩陣, 則對應multi-head 的輸出shape為(ld)xm.

Transformer's Encoder:

  • Transformer's encoder = 6 stacked blocks.
  • 1 encoder block $\approx$1 multi-head attention layer + 1 dense layer.

Transformer's Decoder:

  • Transformer's decoder = 6 stacked blocks.
  • 1 decoder block\(\approx\)multi-head self-attention + multi-head attention + dense layer
  • Input shape: (512 x m, 512 x t), output shape: 512 x t.

Stacked Attention

BERT

  • BERT 是為了預訓練Transformer 的 encoder.
  • 預測mask掉的單詞: 隨即遮擋15%的單詞:

  • 預測下一個句子: 50%隨機抽樣句子或50%下一句, 給予false/true:

相關文章