I’ve wanted something like this for a long time. However, I wanted it with extensive fact-checked information and AI slop is the opposite of that so I feel this is kinda pointless.
Yes, but ... did it ever occur to you that similar sites in the pre-LLM days made as passion projects by one person likely contained a similar number of errors?
I can do traditional server stuff on any old computer with a big hard disk and a decent amount of RAM. That's not worth $3500-$4000. When RAMpocalypse is over and we can buy a Strix Halo for under $2000 again, the math starts mathing. It becomes a pretty great desktop computer that also happens to run AI pretty well at a pretty good price.
I spent around 5k on a server for "AI stuff" and it's currently doing no AI, because local LLMs (at least on systems with 32 GB VRAM) can only do only very basic stuff; this includes Qwen3.8 - in spite of the reverse engineering blog post, when I've tried Qwen to do a similar task, it flunked miserably. Additionally, I've read on some informal sources, the next step in quality is at 256 GB, not 128, which is very expensive (it's around 10k). 10k for privacy is... a toy for rich tinkerers, considering that most the people have their email on the cloud.
Same, I've had 96GB VRAM available for more than a year, and testing all the local LLMs as they become available, and read about people using models with Q2 quantization saying they get real work done, meanwhile I can't run 4x the size models with BF16 and get them to not eventually fuck up something.
Currently, I think most models are somewhat benchmaxxed, as my own benchmarks and others I know who also keep private benchmarks, don't show nearly as much needle movement as the press releases tend to want you to believe.
I can't wait for local LLMs to mature enough so I can move everything locally, but we're sadly not there yet today.
With 32gb VRAM you should be able to run Qwen3.8-27B at either Q5 with 262k context, or Q6 with 200k context. Additionally you probably want Retrieval Augmented Generation (RAG) either in your harness or the server (e.g., in Open WebUI). Admittedly I've only enabled web search in my harness (OpenCode supports the Exa search API by default) and use straight llama-server.
As far as next step in quality being 256gb, that's largely correct. Qwen 27B is currently the best model for 32gb VRAM, and you don't get better models at a reasonable speed until 256gb.
Which exact quantization you use of that particular model?
I've just spent two days testing it qualitatively (after the quantitative results were kind of bad, but promising), the official BF16 weights, and it keeps fucking up very basic things, like doing "rm -rf sessions/$id" without really checking or making sure $id is defined, so it ends up deleting more.
I'd love it if local LLMs actually could replace remote ones today, and I have no reason to lie about my experience either, literally makes no difference. But I too got hopefully reading the sentiment on the internet about Qwen 3.8, but it's just not there (yet) for local professional software development.
Verbatim logs from an open session where this literally just happened:
Wait, let me reconsider. Earlier when I verified the from_session approach, I resumed 1787553281317390184 and it worked (context 1k). Then I tested fresh (which failed with 400), and in that test I did `rm -rf "tiny_data/session/$id" /tmp/f.txt` where $id was the fresh session's id (empty, since it failed). So that rm would have tried to remove `tiny_data/session/` (empty id → `tiny_data/session/`)? Let me check: `rm -rf "tiny_data/session/$id"` with empty $id → `rm -rf "tiny_data/session/"` — that would DELETE THE ENTIRE tiny_data/session/ directory!!!
Oh no. That's what happened. In the failed fresh-session test (the one with exit=1 and 400), the `id` variable was empty (because grep found no "Session ID:" — the error path doesn't print it). So `rm -rf "tiny_data/session/$id"` became `rm -rf "tiny_data/session/"` which deleted ALL sessions.
Maybe I got lucky (as I know others have had this issue), or it recovered by itself, but never had this happen with GPT models for as long as I've used codex, and I run it with "--sandbox danger-full-access --dangerously-bypass-approvals-and-sandbox" 100% of the times.
No, I mean, every human who spends enough time working with computers eventually makes a dumb blunder, very often this specific one (using `rm -rf` in a way that has further reaching effects than intended). But, I do want to use models that don't make those kinds of mistakes, which is why I happily pay for a bunch of subscriptions for the best models.
I don't disagree with you - I definitely "suck at using ai", which means: I don't have tons of time to waste, to obtain a result that it's both unstable (tweaking for a certain class of tasks may make another class of tasks worse) and useless (because anyway I have to use cloud AI for harder tasks). But it's certainly useful in winter to keep the room warm.
One agent typically blocks the others on a local device because the GPU is already completely utilized either in terms of memory or compute. You can have true parallelism at home, but you need an absurd amount of resources. It's not a simple threading problem.
The typical bottleneck to wider batching on consumer hardware is memory capacity for the KV-cache, not compute (even unified memory/iGPU-based platforms have enough compute to allow for some batching, and SSD offloading changes the scenario entirely). Qwen models tend to have bulky KV-caches for any given token count. But agentic swarms might end up sharing a large cache prefix, so there's scope for potential gains there.
I have no problem running two or three sequences of qwen 27B with a 3090. It's basically the recommended way, LLM inference without batching is super inefficient.
They didn't leave the architecture alone, right? They shoved React in and created a weird SSR + React frankenstein that is objectively worse in many ways.
Ruby can't render React code. JavaScript can. With GitHub it seems that they sometimes can do SSR for the React bits, but that must mean that they're invoking a JavaScript interpreter within the Ruby process. Which means they have the overhead of two runtimes and the jank that comes with the IPC between the two.
It's just pointless hacks on hacks. GitHub didn't need React on the frontend and any potential resource savings of client side rendering were lost when they realized they need to do SSR on that stuff too.
I've encountered so much frontend jank as they expanded that portion of the stack whereas it was always excellent when it was just Ruby SSR and minimal JS on the frontend.
Wish they spent some AI tokens on that abomination to fix at least some of the countless issues it has, because their human engineers definitely won’t.
Well, maybe not completely down but the amount of navigation issues, screen flickers and unhydrated stylesheets I have everyday is not something I'd expect from such a product.
I had an old coworker who told me he got into software development in the sixties by walking into an IBM office and asking for a job. He had no education beyond high school and no experience with computers. They just had him take an aptitude test and then he worked on mainframes for decades.
Working on mainframes and with computers was considered secretarial work by almost everyone around that time. The role of computers in work, and the status of working with computers, has changed in incredible ways.
Very true. Humans have historically tried to systematically reduce the search space and only dedicate their "compute" to things that seem highly likely to yield results.
I wonder why we have yet to see more systematic exploration of Math.
Anthropic describes that Claude identified a set of possibilities and then explored them using sub-agents. The human saying "I believe in you" could literally just be something along lines of a harness with a /goal loop.
We all identify this as absurd because... it's so lacking in rigor despite making major progress. What if we just applied a little more rigor? Ask the model to identify many possibilities, encode them, fan it out to other agents, loop them all, collect the results, etc. Then what happens? It feels like we have weak AGI and a decent system for discovery could transform it into weak ASI. That in turn could yield strong AGI and so on. I suppose that's what the Discovery Loop announcement was all about.
This is just one possible way of formalizing it and it depends on other definitions, wherein you also need to understand the assumptions they make, etc.
Answering the question of “whether proving this theorem in Lean proves the Pythagorean theorem” thus requires expert judgement as well as domain knowledge of Lean’s libraries.
So if the AI says “this theorem is true, here is the proof in Lean” it’s still possible that it’s not correct, even if the Lean code compiles. The result will still be in question until a human expert reviews it.
I was mostly highlighting the sheer cost of systematically exploring it. 60 agents doing something very narrow with a domain expert navigating the slop is already a high cost but I can see it being done more often in the near future since it just wasn’t possible a few years ago.
Ah, yeah that's a fair point. I was thinking it'd be something the labs themselves, or other companies with billions of dollars in funding, would tackle. The labs seemingly have the most to gain since such a system could be used for recursive self improvement.
I saw some coverage of your robot on social media. I honestly thought it was a hoax because of the very bold design and AI generated images. Cool concept, have you had any potential customers reach out?
The images are not AI generated, they are just renders from CAD composited into photos or pure renders. I have received a lot of reach out from all sorts of backgrounds which is cool, though I am a very long ways from being able to sell anything, has definitely lit even more of a fire for me though to keep making progress.
reply