The Tokenizers Summary: [EOS],[BOS],[CLS],[SEP]

ForHHeart發表於2024-03-21
原文網址 : https://www.cnblogs.com/forhheart/p/18086534

Resource 1: [Summary of tokenizers](https://huggingface.co/docs/transformers/tokenizer_summary | Hugging Face)
Resource 2: Do you need to put EOS and BOS tokens in autoencoder transformers? | StackOverflow

相關文章