subtopic

Sub-word tokens, padding and attention masks

Learn how Transformers read text: the tokenizer splits words into sub-word pieces ('unbelievable' → 'un', '##believ', '##able'), maps them to IDs, adds special tokens ([CLS], [SEP]), pads short texts to the same length and creates an attention mask (1 = real token, 0 = padding). Texts longer than the limit (e.g. 512 tokens) are cut (truncation).