I've been working on an agent control plane, project management, and version control all in one. I have been building on top of OpenAI's Symphony for a while now, and it finally came to a head where it was becoming too difficult to manage. So, I'm now working on this - hoping to solve agent logistics and ambiguous ownership (read: agents taking over the docs and making everything fuzzy).
I'd love to know if anyone else is working on this and what they've found so far
I guess this "limited set of organizations" is just the standard now. It's just incredibly deflating to see my future as a second class citizen has already come
Brother they can't even release the announcement post cleanly without it constantly going down, they certainly wouldn't be able to release this new model without doing so in stages.
They simply refuse my applications to slightly less restricted models without any explanations. And the current ones refuse automatically to work with me on my papers as soon as they see the word "epidemiology".
When Open AI announced that Astra was the first to reach the "Critical" level in cybersecurity it also said that advanced cyber capabilities are initially provided to a narrow circle of alpha testers like the US government and trusted organizations that Open AI doesn't name. To my mind the "Critical" level itself is an internal scale of Open AI its own Preparedness Framework and not an external audit.
Material wealth is only a single type of wealth. Who's better off - the rich guy who's always yearning to be richer and never satisfied, or the lower income guy that mostly just cares about time with his family and is really happy where he's at?
It hasn't always been the case. Even then, having piles of money still does not gain access to the best military equipment. Sure, we've been living in a time where a couple people get to enjoy a wildly different lifestyle than the average, it just feels like it's about to be different in a way that isn't as ignore-able as someone enjoying a pina colada in a yacht somewhere
Mythos was never released. It's really just the writing on the wall. I'm not going to give up hope, but it's pretty hard to win a race when some people get a jump on the gun.
Being strongly on the AI saftey side of things what is happening was 100% predictable.
At first the race wouldn't even be noticeable. Then people would see things speeding up, for example hardware getting more expensive. Then when the capabilities really got useful most people suddenly realize the race is moving 1000 mph and they are never going to catch up.
I have a pretty good use case for gpt-oss. The amount of time savings has actually been wild. Definitely worth a try. Just to be clear, it gets like 2000tok/s
2 hours in and this thread is already stacked, but I'll bite since I am stuck on this problem and need help. I am working on a language learning solution that involves llms. The way I am branding it is "Anki meets Ai" because it combines a flashcard-esque method of generating complete exercises such as multiple choice, cloze, etc. with the tried-and-true SRS methodology.
I think it works great! The problem is, I think it works great. The issue is that it is doubly-lossy in that llms aren't perfect and translating from one language to another isn't perfect either. So the struggle here is in trusting the llm (because it's the only tool good enough for the job other than humans) while trying to look for solid ground so that users feel like they are moving forward and not astray.
- LLMs are better at critiquing translations than producing them (even when thinking, which doesn't actually help!)
- When they make mistakes, the mistakes tend to be different to each other.
So it translates with the top 4-5 models based on my research, then has another model critique, compare, and combine.
It's more expensive than any one model, but it isn't super expensive. The main issue is that it's quite slow. Anyway, hopefully it's useful, and hopefully the data is useful too. Feel free to email/reply if you have any questions/ideas for tests etc.
Sorry, when you said hybrid I was expecting something that was partly an llm and partly something else. How did you arrive at your coherence/idiomaticity/accuracy numbers (if you'll forgive me not delving too deep into the website)?
Hybrid as in a combination of different LLMs. I recommend trying the demo on the site, it should give you an idea of what it's doing. The code is also pretty short.
So those numbers are from an older version of the benchmark.
Coherence is done by:
- Translating English, to the target language, to English
- repeating three times
- Having 3 LLMs score how close the original English is to the new English
I like it because it's robust against LLM bias, but it obviously isn't exact, and I found that after a certain point it's actually negatively correlated with quality, because it incentivises literal, word by word translations.
Accuracy and Idiomaticity are based on asking the judge LLMs to rate by how accurate / idiomatic the translations are. I mostly focused on idiomaticity, as it was the differentiator at the upper end.
The new benchmark has gone through a few iterations, and I'm still not super happy with it. Now it's just based on LLM scoring (this time 0-100), but with better stats, prompting, etc. I've still done some small scale tests on coherence, and I did some more today that I haven't published yet, and again they have DeepL and Lingvanex doing well because they tend towards quite rigid translations over idiomatic ones. Claude 4 is also interestingly doing quite well on those metrics.
I need to sleep, but I can discuss it more tomorrow, if you'd like.
I'd love to know if anyone else is working on this and what they've found so far