Right, other words see higher attention as it builds a sentence, leading it towards where it “wants” to go, but LLMs literally take a series of words, then spit out then next one. There’s a lot more going on under the hood as you said, but fundamentally that is the algorithm. Repeat that over and over, and you get a sentence.

If it’s writing a poem about flowers and ends the first part on “As the wind blows,” sure as shit “rose” is going to have significant attention within the model, even if that isn’t the immediate next word, as well as words that are strongly associated with it to build the bridge.