I’m sure there are a wide variety of experiences out there, but here’s my perspective as a former biglaw associate and current solo litigator:
I have had some success using frontier models from the last 6ish months, but only when I can break up my work into discrete and verifiable tasks. For example, I had ~15k pages of discovery I needed to dig through for a summary judgment motion. Instead of just asking Claude to find the best evidence, I asked it first to run a clean, high quality OCR pass (it was almost entirely PDFs). Then I had it generate embeddings and write some reusable python scripts to make keyword and semantic searching easy for agents. While I was writing the brief, I would routinely ask my agent (Claude Code) to use both keyword and semantic searching to find the best evidence supporting whatever assertion I was trying to make. I trusted it because there were traces I could follow.
In other cases/situations, I’ve tried just giving a model access to all the docs and saying “write a brief arguing X,” but it’s always terrible at this. It writes briefs with lots of evocative jargon and rhetorical flourish, but a low signal-to-noise ratio.
Again, I’m sure others’ experiences differ based on workflow, legal area, etc.
Agreed. Six months ago, it was basically a gloried grammarly.
But lately, I’ve been taking hints from the “company brain” models, where it develops a running model of the case, and assesses each new piece as it comes in and updates the file.
I’ve also been using “Ralph Wiggum”-type models where you pass letter or contract drafts back and forth between agents with different goals (rules compliance, grammar, conciseness, ai slop detector, an opposing counsel critic, etc.). After a few rounds, it’s not perfect — but I start with a very good first draft in my hands.
In some ways I miss the early LLM days before RL and system prompts (I assume this is the difference) made them all sound the same. They would say weird stuff. They got better at being useful and worse at being fun.*
*This sadly also describes me as a 31 year old man.
A major difference is they weren't being sold as "assistants" back then. Now they're a product expected to act a certain way. While some LLMisms probably come from shared poison in the pretraining corpus, many of them also come about as a result of trying to make them more appealing as a product. Most open weight models are just a company's product, post-instruct and everything. Base models are a different ballgame (and are becoming increasingly sparse), so effectively no matter what you're pulling in someone's model that's been trained into the chirpy "executive assistant" personality we all hate. Some models are worse for it than others.
You can tell the post-training makes all the difference though. Gemini has a different stink to it than DeepSeek.
The false positive rate for Pangram 4 is something like one in 24,000.[0] To put that in perspective, the wrongful-conviction (false positive rate) for death-sentenced defendants in the US is estimated conservatively to be around 4.1%.[1] The FP rate for death-sentence convictions is 1,000 times bigger than Pangram’s FP rate.
Now, the US criminal system is not a great yardstick for justice. But it goes to show you Pangram is really good evidence that something was LLM generated. It can be an amazing tool for enforcing AI policies in schools, and there ought to be ways to use it with caveats for the rare but inevitable false positives (appeals, etc).
There are a couple of statistical errors in your argument here.
First is frequency. Even using Pangram's claimed numbers, the University of Georgia should expect to see several false positives every week. Remember that the metric is # of assignments run through Pangram, not number of students. A campus of 40k students will see many more than 40k assignments every week, and so should expect honest students to be accused of cheating with some high degree of frequency. You're comparing infrequent events (death penalty sentences) to high-frequency events (students submitting assignments).
And obviously, you are citing a company marketing document as fact, of which we should all be suspicious. (There are also obvious problems with the eval dataset that the paper does not address.)
Second, you're using the upper bound for Pangram's claimed numbers and the lower bound cited in the NIH publication.
> at least 4.1% would be exonerated. We conclude that this is a conservative estimate of the proportion of false conviction among death sentences in the United States.
> The false positive rate for Pangram 4 is something like one in 24,000.
Gotta suck to be one of the 8B/24k=~300k people in the world whose writing pattern is falsely labelled as slop by this tool that people say is so accurate so customers are going to feel really sure about your alleged dishonesty about writing your own texts
This false positive rate is a double-edged sword. Please still be careful when accusing people
I don’t mind AI-written code. But an obviously generated README.md is such a turn off. If you don’t take the time to explain what your software does in your own words, I struggle to trust that it’s been thought about much at all.
Yeah, I really think this should be the new universal standard. Mostly for respect reasons, but also going through and saying “what DOES my app actually do?” (which is something writing a README forces you to think about) is pretty much the minimal last line of defense attempt at quality control.
You can and probably should do more quality control than that, but it’s a reasonable assumption that if the initial landing document hasn’t been quality checked, the rest hasn’t either.
If the app is good the app is good. End of discussion
Why do you care how it was made? What’s next? It matters what country or city the developer is from or ahah language it’s written in or what machines were used or how the electric was sourced
The complaining about the pelicans is so strange to me. It’s just a fun heuristic. If something is claimed to be AGI, I’d expect it to be able to make svgs.
Yes, I know what an emdash is---I've been using them in my writing since long before they came to the fore of the AI writing conversation. Anthropic's use of the emdash in the fragment I quoted is clumsy and reads poorly relative to the obvious alternative, a comma.
I didn’t mean to suggest you don’t know what an em dash is. But you said “this isn’t how you use them.” And my response is: actually, this use of them is totally fine.
I have had some success using frontier models from the last 6ish months, but only when I can break up my work into discrete and verifiable tasks. For example, I had ~15k pages of discovery I needed to dig through for a summary judgment motion. Instead of just asking Claude to find the best evidence, I asked it first to run a clean, high quality OCR pass (it was almost entirely PDFs). Then I had it generate embeddings and write some reusable python scripts to make keyword and semantic searching easy for agents. While I was writing the brief, I would routinely ask my agent (Claude Code) to use both keyword and semantic searching to find the best evidence supporting whatever assertion I was trying to make. I trusted it because there were traces I could follow.
In other cases/situations, I’ve tried just giving a model access to all the docs and saying “write a brief arguing X,” but it’s always terrible at this. It writes briefs with lots of evocative jargon and rhetorical flourish, but a low signal-to-noise ratio.
Again, I’m sure others’ experiences differ based on workflow, legal area, etc.
reply