subtopic

Self-attention: Query, Key, Value

Learn the simple recipe: each word makes a Query (what am I looking for?), a Key (what do I contain?) and a Value (what do I pass on). Scores = Query · Key; softmax turns scores into weights; output = weighted sum of Values. Code it in about 10 lines of NumPy on a 4-word sentence.