SciTech-BigDataAIML-LLM
PE(Positional Encoding)位置編碼:
- BOW(Bag of Words)詞袋模型：丟棄Word位置資訊, 只統計Word之間的 Co-occurrence Probability(共現機率)。
- RNN(Recurrent neural networks): 有Word的Position資訊。
- Transformer: Positional Encoding, 將Absolute Position位置資訊Embedding 嵌入 Word Embedding Vector。
- BERT: Trainable Position Embedding.
- GPT: ?
- Latest: Rotate Position(最新的旋轉位置編碼)。
數學公式應用:
-
向量的“Dot-Product Similarity”點積相似度:
$\large Similarity_{ij} = \xRightarrow[ Word_{i}^{T} ]{} \cdot \rightarrow[ Word_{j} ]{} $
\(\large Word_{i}^{\rightarrow} = e_{i}^{\rightarrow} + p_{i}^{\rightarrow}\)
\(\large Word_{j}^{\rightarrow} = e_{j}^{\rightarrow} + p_{j}^{\rightarrow}\)
$\large e_{i}^{\rightarrow}: Word Embedding Vector with "i" as its index $
$\large p_{i}^{\rightarrow}: Positional Embedding Vector with "i" as its index $ -
$\large $
1 Absolute(絕對)Position
2 Relative(相對)Position
