This team is in a tough spot - they'll need to innovate and get their models in a good spot quickly. They are racing the clock against various similarly sized models being burned into ASICs and running at a similar speed (or faster).

They can run it on „commodity” hardware as gpus, that gives them ability to change direction fast without burning money on ASIC? The space moves fast so model on ASIC can be outdated in couple of months?

I think the question is probably more along the lines of - why not do a database with 100 TB of storage/records instead of a cache? tomato / tomato.. especially with smart caching in front of database. 100TB of flash is a good bit cheaper than 100TB of memory

Because it would be slower and have different scaling requirements than the ones they want.

It's not 100TB of data. It's probably 50 GB of data on each of 2000 servers. Because it's a cache. What is the point of a central cache if it's as slow to access as the original data?

TFA gives numbers closer to 5GB.

I'm no expert but presumably all of throughout, latency, and churn. DNS is approximately a giant KV store where the typical record has a TTL of ~5 minutes.

This is smart, task-specific caching in front of database.

I'd take qwen3.6 (3.8 as of tomorrow) 27B running at 17k per second first on the way to Sol/Fable! And then dsv4-flash-0731!

Right! Not sure that will happen since Talaas was just bought by AMD, so more likely a US model. Still, we can dream.

Indeed, you could literally brute-force it that way :)

Opus 4.5 level of performance is also accessible with deepseek-v4-flash-0731 (0731 being the july 31 update) which is much, much, much smaller. 2x RTX pro 6000 blackwell can run it. 4x can run it very comfortably

I am running DS v4 flash 0731 lossless at 80t/s right now. It really is not at Opus 4.5 level (for my workload). I would say it's around 3.7 Sonnet, which is still pretty good, but other models such as GLM 5.2 are still leaps better. Of course I run DSv4 flash over GLM 5.2 for a few very good reasons, but intelligence is not 1 of them.

Despite fitting into VRAM, I can't get DSV4 to run at usable speeds on my AMD hardware. The upcoming qwen3.8 27b greatly excites me, and I hope it can outperform Stepfun 3.7 Flash, which is the best thing I can run today.

I'm just trying out Muse-Glimmer 30b, and my initial vibe is this might be better than qwen3.6-27b. No idea how it compares to Stepfun, because I can't run that model - but worth checking out while you wait for qwen3.8-27b

Anyone thinking of buying 2x RTX Pro 6000 Blackwells - beware: unlike other cards e.g. RTX 5090, The RTX Pro 6000 cards cannot be NV-Linked, so you'll be going through the PCIe bus instead (7x higher sync cost)

My understanding is the last consumer card that supported that was the 3090. A Google search seems to agree the 5090 does NOT support NVLink...

What do you need the extra 2 for? Tensor parallelism?

Longer context and more cache. The problem is that native format with DSpark enabled you have very little room on the VRAM.

I was under the impression that you could fit the full 1M context within the 192GB VRAM as a result of DeepSeek's various architectural advancements, but I'll grant that DSpark + a larger pool for concurrency may necessitate more VRAM, yes.

Modern dell desktops (like 8th gen+) idle around 10W. Raspberry pi isn’t saving much

I’m impressed with that.

I had a Nuc 9 extreme with a 10gb SFP card in it.

It averaged 69w. The Mac Mini I replaced it with uses just under 5w.

A Mini is possible the worst bang for buck. Also, the horror that that is Desktop Docker can’t be over emphasised. I know there are better ways of using Docker on a Mac, but having got something working I can’t face rebuilding it.

My home router is a Lenovo ThinkCentre Tiny with a Core i3-8100T with a dual 10Gbit SFP in it (1x DAC, 1x 10GBase-T) and it idles at 14W, routing 10gbit full bore 24W. I got it for $80

I also have some Fujitsu office machine I picked up for $50 as my "I need to run some random task on a physical linux box" machine with an i5-7500 and it idles at 15W

Some of those SFP cards can use quite a bit of energy on their own. They're not designed with energy efficiency in mind.

I've got an Optiplex with an i5 and two hard drives and that idles around 12W when idling.

But when starting from scratch, it might be better to get an older Mini that is supported by Asahi and run Linux on it.

What's wrong with Docker desktop? Been using it everyday for at least 5 years.

It’s sluggish, it’s incredibly bloated with all the services that Docker (the company) wants to sell you and the UI is badly designed.

OrbStack is my go-to when it comes to running docker on Mac. It also uses macOS native containerization APIs so containers will be a bit more performant than they would (at least use to) be on Docker Desktop.

Another good approach if you’re fine without a GUI or want to bring your own is just ‚brew install docker‘.

The main gotcha I encountered with orbstack was with pulling down internally built amd64 images and trying to run them on apple silicon. I just remember constantly running into issues and going back to docker desktop as a result, even if their solution is janky in its own way.

I'll check it out. Yes, no UI needed. I run Docker Desktop but I don't think I've used the UI more than on ce.

Trying to get it to open on startup and start containers in a particular order is a little finicky. But the real hell is trying to get it to start containers after the network shares have connected.

I’ve got it working, but compared to doing it in compose, it’s awful.

Not much, it's an "I use arch btw" kind of thing but for docker.

I made calculations and got the same results. Bought low profile and as a server

“Search session history for when we discussed XYZ”

https://github.com/nerdyaustin/memory_mcp

You can skip all the sync stuff, not necessary at all

which would make it the 37th+ spacecraft to hit the moon

(most S-IVB rocket stages and Lunar Module ascent stages impacted the Moon - not tracked individually on that page)

https://en.wikipedia.org/wiki/Category:Spacecraft_that_impac...

Well when a huge part of potential revenue is all in on Bedrock... you need the harness to be able to talk to Bedrock. And Vertex. And all the other places these models are hosted. And allow for proxy because many businesses do not all direct internet access... all valid business reasons.

This is/was classic market segmentation. Want ECC? Pay up for Xeon (server/workstation CPU)

AMD has historically supported ECC on all it's chips. That's changed in the last gen or two where some of the lower end APUs don't support ECC but the Pro APUs do and all the ryzen stuff (5, 7, 9) support ECC if you get a motherboard that supports it.

Theoretically, though I’ve not seen any certified builds. At least advertised. Do tell who offers an ECC laptop PC at a reasonable price.

Laptops are pretty much unavailable with ECC especially with soldered memory becoming common. On the desktop side, there's multiple motherboards that explicitly note that they support ECC memory. AsRock and Asus both include ECC memory on their QVL lists. I'm not sure how many ITX boards support ECC though so getting it on something that's not a tower might be tough.

But it's definitely possible, I have an ECC setup on my desktop and it's caught a memory error or two over the last few years.

Even with the desktop I want to see “ECC guaranteed” on the box, not a pile of parts that “should” work.

It's hard to get more ECC guaranteed than the mobo manufacturer having ECC ram on their QVL (qualified vendor list) for the motherboard and indicating that ECC is supported in the specs. I suppose you can just get a complete workstation with ECC from HP or DELL and that'll be guaranteed to work but you'll certainly pay a hefty premium for that.

Sort of, often there's workstation chips, like the xeon E3, which is a warmed over desktop chip ... with ECC. Price difference when I bought was $20 CHEAPER for the CPU (at the cost of 100 MHz or so). I did pay $50 more for a motherboard and 10-15% more for ram.

Or just get an AMD Ryzen.