> It works in precisely the same way that you can walk from "here" to "there" by taking a step towards "there", and then repeating.

It's funny how, in order to explain one complex phenomenon, you took an even more complex phenomenon as if it somehow simplifies it.

Sorry, can't tell if that's sarcasm or not.

I wasn't referring to the biomechanical process of walking, I was referring to the process of gradient descent, which is well understood and yes, quite simple.

If that was true, knowing how elementary particles work would give us understanding of the whole universe, in which case no other science would exist. But other sciences do exist, ergo, you're wrong.

That's an interesting perspective.

Thing is, that would be a rebuttal if I'd said something about the underlying tensor fiddling code being understood, and you were claiming that next token prediction was a mysterious emergent phenomena.

Unfortunately that's not the argument I made. My claim is that there's nothing surprising or mysterious about the fact that a system designed to repeatedly generate a highly likely continuation of a sequence of tokens (considered as a member of pre-specified class of sequences) winds up producing something that looks like it could be a member of that class. That's kind of the whole point. These things were designed to predict plausible next tokens, and that's exactly what they do.