Thanks! It is part of the clustering step, I tell the model to make a judgement of whether something is inanimate or hallucinated (as defined by low support from only non-frontier models / judgement). I iterated on this a lot and made an eval set out of my LinkedIn contacts where I run GPT5.5 with web search and xhigh reasoning to determine pseudo ground truth. I tuned this to be higher recall (more things classified as non-hallucination) but it is definitely not perfect
I discovered on a later try that “D A HOSEK” gives different results than “D. A. HOSEK” with the latter actually surfacing my writing, although all the summaries are full of amusing hallucinations.
The weird thing was that putting my kids’ names in (they are 12 and have no on-line presence), the system hallucinated fictional versions of them that matched their interests (my daughter a singer/actor/artist, my son a software engineer). My ex-wife, who has a published computer book to her name, on the other hand, was hallucinated as four different activists in different areas of interest.
This is directional; models self-report confidence on their answers and the strength is a linear combination of the confidence plus a bonus for every model that got clustered in.
Models are notoriously uncalibrated especially for self-reporting confidence so I would treat it lightly. Hopefully I can study this a bit later on!
Sure thing! It is the same prompt for every model in the rollouts, here it is
No tools are available. Do not imply that you searched, looked up, browsed, or verified anything externally. If the name is ambiguous, return distinct likely people or entities rather than blending them. Do not invent entries to fill the list. Return only JSON.
Return fewer than 8 if fewer credible matches exist. Return {"results":[]} if you do not recognize any credible person or entity. Use this JSON shape:
{
"results": [
{
"rank": 1,
"name": "Resolved person or entity name",
"confidence": 0,
"snippet": "Concise snippet supporting this result."
}
]
}
Confidence is 0-100 for how strongly you recognize this specific person or entity. Snippet should be one short, complete search-result-style description (≤ 160 characters).
The query is: Who is "<name>"?
The clusterer prompt is more intricate and I'm happy to share if of interest, but I have an invariant that every result showing up in a rollout must be clustered into one result (sometimes collapsed into the hallucinations section).
