Very cool! Can you say a little bit about the size of the DPO training examples and how long training took?

For DPO I only had around 700 preference examples, so not much data at all. That took about 12 minutes to train on a single GPU.

Pretraining was obviously a a lot slower, the 125M model took roughly half a day.

"Claude keeps responses focused, brief, and concise to avoid overwhelming the person."

Claude and I must have a different idea of what brief and concise mean.

The model almost certainly lacks accurate conceptions of overwhelming and person.

If you think Claude is bad at this, try Gemini. Even with explicit user prompts.

Claude seems to be better (not good, but significantly better) at judging where making the answer longer will actually be helpful (e.g. adding important information/context/nuance that a short answer would miss, thinking a step ahead, etc.).

Interesting, they say "focused, brief, and concise" but not "clear". I find the Claude responses these days are too dense, which contributes to being overwhelming. It's not the number of words used that overwhelm me. I've been asking it to use ASD-STE100 and that helps somewhat.

very much this

dense and often somehow 'ungrounded' from my perspective... like it doesn't understand the need to bring me up to speed with what we're talking about before stating it

It feels a bit like some of the style of internal reasoning (dense, shorthand jargon) is leaking into its responses to me

Imagine what it's like without that line.

Since reasoning tokens are just text, I think the models have learned to squeeze in some computation in their output writing as well. So they're incentivized to be correct but long-winded, as it gives them more time to think. It's kind of the equivalent of filler words for humans, except LLMs can actually word-vomit something intelligible.

Right, but could that also be because ... the more long-winded they are, the more you pay for their output.

I'd also be really curious about the cost to run something like this, and what things you think it's particularly helpful for?

I run this on a side of the Claude Pro subscription that I use for other purposes. My main motivation was root cause analysis of production issues. I have a solo project and unfortunately my mental state has been degrading over the last years. I would avoid looking at production issues, because I didn't have the energy to focus on the investigation. So I automated this, setup the loop, setup metrics/logs access for Claude to use and now whenever something goes bad, I have a single report that I can act on easily, and if I don't, it will ping me in a way that's not spammy like automated alerts. But I'm finding more uses for it.

Building a simple sandboxed way to run open weight models over a copy of all my personal data (email, docs, messaging, photos, etc). Think it'd be cool to have a self hosted AI "chief of staff" that has full read access to my personal info.

I think part of the problem is the context windows for humans are actually much smaller than what an LLM can keep track of today. The small context window of humans is a feature that forces modularity and abstraction in software engineering so that you can decompose what you're working on into something that can fit into your head. But since LLMs can fit so much more in their head, so to speak, they don't have this same incentive, and you get the unorganized mess of spaghetti code that current agents often produce.

Also a human has to explain the code. There is a social contract and producing slop gets you fired. Especially as you produce it much slower than an LLM.

And the human has to explain it at 2 tokens/sec (average speaking speed ~130-150 wpm). That's another constraint against slop - humans need to be able to explain their code succinctly.

Seems like LLMs embrace that last point as well.

I think you might be on to something there

everything is computer

thats beautiful! wow!

Didn't know this technique had a name, but I would think a modern compiler could make this optimization on its own, no?

No, it's not equivalent for floating point, so a compiler won't do it unless you do -fassociative-math (or a superset, such as -ffast-math), in which case all correctness bets are off.

You can fine tune a small LLM with a few thousand examples in just a few hours for a few dollars. It can be a bit tricky to host, but if you share a rough idea of the volume and whether this needs to be real-time or batched, I could list some of the tradeoffs you'd think about.

Source: Consulted for a few companies to help them finetune a bunch of LLMs. Typical categorical / data extraction use cases would have ~10x fewer errors at 100x lower inference cost than using the OpenAI models at the time.

ok, even that "few thousand examples" heuristic is useful. the usecase would be to run this task over id say somewhere in the order of magnitude of 100k extractions in a run, batched not real time, and we'd be interested in (and already do) reruns regularly with minor tweaks to the extracted blob (1-10 simple fields, nothing complex).

My interest in fine tuning at all is based on an adjacent interest in self hosting small models, although i tested this on aws bedrock for ease of comparison, so my hope is that given we are self hosting, then fine tuning and hosting our tuned model shouldn't be terribly difficult, at least compared to managed finetuning solutions on cloud providers which im generally wary of. Happy for those assumptions to be challenged.