I tried getting LLMs to add proper Vulkan support to ik_llama.cpp, which have very good support for CUDA and CPU. The models do an admirable job; they don't care much about poor DX.
Few problems I noticed:
* coopmat2 from nvidia is the classic embrace, extend, extinguish. No point to ask the models to translate from CUDA to coopmat2. Instead, the models can understand the existing CUDA and CPU kernels, and adapt them accordingly to non-nvidia devices.
* However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex. Upfront standard API might just be an evolution dead end.
On the other hand, despite my complain about the standard API, the models were able to come up with cooptmat1 kernels that run dsv4 flash faster than whatever the guys at antirez/ds4 can come up with using rocm, on a strix halo, with the added benefit that I can also pair the strix halo with an egpu to drastically speed things up.
From what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across devices. It is fair to say that nobody would replace cuda with coopmat2? That looks like a EEE project that can assigned to a couple of nvidia engineers, to fragment the ecosystem.
coopmat2 features will eventually be rolled elsewhere. coopmat also started as an NVIDIA extension.
The client use cases that coopmat was intended for are customer machines, not multi-GPU, which is broadly seen as a datacenter feature instead. That said coopmat orthogonal to this.
So when I said "a couple of nvidia engineers", I indeed meant Jeff.
VK_KHR_cooperative_matrix - embrace?
VK_NV_cooperative_matrix2 - extend?
I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
> I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s
> Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable
__VK_ENABLE_DEVICE_GROUPS=1.
> $20k workstation, best case: $15k M5 Ultra 512GB, 36-month amortization, ~$440/mo. Runs a GLM-5.3-class model at ~30 tok/s. Saturated 24/7 it produces roughly 58M output tokens/month.
For agentic coding, ~90% of the cost comes from cached input tokens. This cost increases quadratically with the session length. If sessions go near 1M context, the number of cached input tokens can easily exceed 1B in a day.
> Both the NVFP4 and AWQ W4A16 failed to properly close their tool calls ...
If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once a tool call is detected.
> ... and botched Cisco command line syntax (the correct command was ‘show arp’, while they executed ‘show run’)
But this failure mode can still happen.
Anyway, NVFP4 and AWQ W4A16 are generally regarded as low quality quants. IQK/Trellis quants from ik_llama.cpp and EXL3 quants from exllama should work better.
So, perhaps the lesson here is "don't use vllm at home"?
Let's say DeepSeek is being forced to use the CANN stack, and the new pricing reflects the cost when 100% of inference is done with Huawei chips. Then, I suppose we can infer that:
* CANN stack is 1.5x~2.3x less efficient in compute
* CANN stack has 6x lower inter-connect capacity
> computer chips once again become a commodity
Ascend 950 is going for $7k to $9k with mediocre looking specs. $16k for RTX Pro 6000, $6k for RTX Pro 5000. This is not looking good.
It would have to be reproduced on a CUDA stack because AFAIK Huawei don't sell the Ascend 950PR (for inference) to anyone, rather, they operate them as part of the Huawei Cloud and only allow select customers (such as DeepSeek) to rent them.
The CEO of DeepSeek recently revealed to investors a lot about the resources available to DeepSeek, and the gap between Huawei and NVIDIA. Select quotes from the transcript (translation is a bit patchy on the source website though):
"We currently have roughly 20,000 H-equivalent compute cards"
"Huawei 950—right now Huawei gives us 16,000 cards, this should be publicly stateable."
"Like Huawei gives us roughly 16,000 cards of capacity, internet giants maybe get a hundred-something thousand, we get ten-something thousand—I think this ratio is also relatively... but this is probably just how much capacity Huawei has."
"16,000 Huawei 950 cards only equal 4,000 B-series cards."
"Huawei’s supernode, Huawei’s 950 supernode, in performance and price can completely substitute for NVIDIA’s GB200, GB300. The price is definitely more expensive, but limitedly so. Fifty percent more expensive, a hundred percent more expensive—a hundred percent more doesn’t matter, two hundred percent more doesn’t matter. For example, a hundred percent more expensive—I think it can already be considered a price-level substitute."
"I think domestic hardware might need a few years."
"I don’t quite believe that five years from now, we’ll still be stuck on the production capacity problem. Right now we’re definitely stuck on the production capacity problem—this year, next year, the year after, I think we might still be stuck on the production capacity problem, but five years later, I think maybe not necessarily—I’m still relatively optimistic."
Objectively speaking, the 2 bit quant from antirez has very low accuracy. Meanwhile, his 4 bit quant does have decent accuracy, but is a bit pointless by being bigger than the full precision MXFP4 quant. Anyway, they all work fine in practice.
Few problems I noticed:
* coopmat2 from nvidia is the classic embrace, extend, extinguish. No point to ask the models to translate from CUDA to coopmat2. Instead, the models can understand the existing CUDA and CPU kernels, and adapt them accordingly to non-nvidia devices.
* However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex. Upfront standard API might just be an evolution dead end.
reply