True, but the ad is to my own product, there are no advertisers, maybe there will be at some point, but if they were, that would barely cover the costs of testing the models, and likely never get a ROI on the hundreds of hours I've spent building it.
Would I want it to grow and make money at some point? Sure, why not, then I can test even more models at higher reasoning levels. Meanwhile it's just me testing models when they come out and publishing the results for anyone who finds them useful.
I don't see why posting some info and a link with my own findings, in a relevant discussion is considered spam. Should it be?
Consider adding a rule that an author must disclose (in their own words) for what parts and to what extent LLMs have been used to assist their project.
I don't think it would help: most people wouldn't know about it, of those who did many wouldn't conform, and there would be no reliable way to enforce it.
Can I ask what exactly you are building? Your experience tracks for me when building a real product -- something I want other people to use. Most of my time on these projects is spent talking to my users and carefully refining my requirements and design.
For personal pet projects I can definitely see how you can blow through your token budget very quickly. If I just point my coding agent to iteratively come up with some heuristics for some NP-hard problem, it will read intermediary outputs and constantly make small changes "in the dark" until it either finds a small improvement or gives up. In a similar vein I found that you can burn many many tokens if you try to let the agent reverse engineer something where you don't have the source code. If you just give it a binary or some interface to work with and a vague task you can easily burn your entire budget with 1 prompt.
I wouldn't want anyone to use these fully vibe coded toy projects though; it is more of an exploratory curiosity for me where I learn more about some problems I'm interested in as well as gauge how good the agents are at tasks that I seem to have a much better intuition on how to approach.
I'm building a website that allows friends to write branching fiction novels together, and another website that allows people to argue and conclude together using first-principles thinking. I've abandoned a project that allows people to register percentage certainty of sports outcomes to improve their calibration - it wasn't fun enough for the users. A few other things besides. I recently wrote a gpl TUI to edit dags here: https://github.com/tunesmith/dagim - that one as something of an experiment since it's in a language I don't know, that one took 3-4 days of steady prompting.
Very interesting, thanks for your reply. I don't have much domain knowledge with your sorts of project so I don't have an intuition on how well it would go with coding agents. I have done quite a lot of programming with graphs, but I don't understand the motivation of the tool you linked. From the video included in the repo I can see an example for cooking, but from my own experience I can hold this size of a graph well enough in my head to make optimal decisions in real time. And for bigger graphs I never had to create them by hand, but rather from datasets and then make adjustments to the graph with some heuristics depending on the problem. I don't understand the use case, at least for myself.
Cynical take: If their model was so groundbreaking they wouldn't have to involve the government for their marketing campaign. You would notice shit breaking everywhere; oh wait, how many days has it been since the last supply chain attack? What was advertised with Mythos was already possible with 2024 LLMs if you had some basic hacking knowledge.
Thanks for documenting your personal observations. I do have a few questions. First, could you expand by giving other examples on how you observed this model to be relentlessly proactive?
From my personal experience with prior frontier models using both Claude Code and Codex I found them to already be quite proactive depending on the domain (although Codex a bit less so, which I personally prefer).
The main task that they seemed to struggle with for me are tasks that naturally have long run times for the programs the agents wrote, as they didn't seem to have a good intuition for when/how to change approach to minimise the time spent on the task. Specificically if you are trying to scrape sites/services that are heavily guarded against programmatic access or running automated tasks that call LLMs (such as indexing or document extraction).
I'm not surprised that for web dev the proactiveness is the most obvious improvement, as I would expect the most common use case with the most training data to be the biggest priority. I have previously built a similar workflow as you described Fable 5 to auto test changes to the website and while it worked somewhat well, it often couldn't identify obvious flaws to the human eye, such as overlapping text or inconsistent font choices as well as bad layout decisions. I do like it for quick prototyping, but the testing and design decisions were not ones I would hand off at this moment.
Did you notice improvements in these areas? Can you share how it does for long running programs?
If you want I can give you some more specific instructions to test, but I would also be happy to hear from your own use cases.
The visual regression point is interesting. In my experience, the models that do best at "overlapping text/bad layout" catches are the ones being fed actual screenshots rather than DOM snapshots. If Fable is doing screenshot-based diffs natively, that would explain an improvement there, but I haven't verified it.
From how Simon described it it's not a native feature, but one that the model built as a solution for automatically testing. You could already instruct the agent to write a program that saves screenshots to disk and then reads it. As long as the model is multimodal (which pretty much all releases are these days) it can "natively" interpret images. There's probably a clever way to engineer this to be somewhat efficient, but for me it was rather token hungry, because the testing inputs and the description are usually quite verbose. I suppose you could use a weaker model for navigating the test and then only feed the output to the stronger model.
If you upgrade your 8 year old phone the many incremental upgrades will be very noticeable. From my personal experience the LLM space is also moving at a faster pace than the phone industry at the moment, but at least from a financial perspective I would expect it to slow down sooner rather than later.
Linguistics, specifically as it pertains to language learning
Edit: Whoops read your question wrong. I do a bunch of NLP on different languages, and use LLMs to pad out and interpret the data. Asking for things like translations, alternatives, transliterations; associating and validating data; transferring data from one language to another; segmentation and cross lingual alignment; the list goes on.
I did manage to get higher quality in the end, so it’s not entirely a regression. But older LLMs were much more capable with less prompting at interpreting disparate data and tying it together.
Most of the work I do does not really have a “right answer,” just a lot of wrong ones, which I think is what trips up LLMs. If I turn on reasoning for any step in my pipeline, the token count goes up 100 fold and the quality gets cut in half.
Edit 2: I did have to move off of GPT though to get the improvements mentioned. Go mistral!
What kind of data are you interpreting? Do you mean document extraction from different languages? I have only used GPT5.5 for agentic coding, which did get significantly better from my experience, although that does align with your conjecture of their focus being on improving this. I haven't noticed a regression when it comes to interacting with it in different languages though (specifically German and Russian). I have done data extraction from documents in different languages, but only with locally hosted LLMs (mainly Qwen3.5-397b) as I cannot legally use cloud-based solutions. My local solution was more than sufficient, so I would be surprised if a frontier model would fail at that.
