Yes, it would have to be unable to reason about its own confidence levels, wouldn’t it? It produces content—and, as far as I understand, sort of simulates basic reasoning—by making predictions based on a huge corpus of text. The larger this corpus becomes, and the more sophisticated its method of analyzing it, the better it becomes at “reasoning about” the things described in its corpus.
But the question “How sure are you?” inherently refers to something—the LLM’s “mental state”, if such a thing can be said to exist—that isn’t referred to anywhere in the corpus. No improvement in the quality of the corpus or the power of the predictions made based on the corpus can have any impact on this problem.
But the question “How sure are you?” inherently refers to something—the LLM’s “mental state”, if such a thing can be said to exist—that isn’t referred to anywhere in the corpus. No improvement in the quality of the corpus or the power of the predictions made based on the corpus can have any impact on this problem.