Apple needs better GPUs for "AAA gaming on macOS". My M2 Max is pretty awful at games that run fine on a desktop GPU. An M(whatever) Ultra to play games seems like a debatable value proposition.
Having rolled my own RAG the other week, I personally would recommend talking through it with Claude Opus or a similar model.
My baseline was (vibe coded) full-text search with SQLite, and we landed on long chunks with overlap, with a really simple nearest-neighbor search: quantize the index to a sign bit per scalar, which makes it super cheap to estimate dot products, then calculate better (8-bit quant index times native precision for the query) dot products to sort the top documents. A vector database would make sense for a much larger corpus, but I currently have fewer than two million rows. Claude vibe-coded it to use an OpenAI-speaking local inference server and made semantic search an optional augmentation for the full-text search.
I still don't understand this. Whenever I ask a model how to go this they recommend rolling my own. Is this really too niche to be readily available and well maintained in a GitHub but also so common every LLM knows countless ways to do it?
LLMs do tend to reinvent the wheel. There are off-the-shelf solutions. In-process, the major options seem to be Meta's Faiss, Spotify's Annoy and Alibaba's zvec. zvec's shared libraries are tens of megabytes in size, and I haven't looked closely at the others. My program is in Go, so even C bindings can be a pain.
If you have billions of vectors, you should use a dedicated implementation: nearest-neighbor algorithms in high-dimensional space can be tricky, and there are a lot of trade-offs. My case is amenable to a simple implementation because I don't need huge scale. That's also why I did not need or want an out-of-process server.
A lot of the article focuses on problems induced by a 512-token input limit. For example, one needs a lot more chunks with such a small input, especially with overlap. I realize that some embedding models do have input contexts that small, but 8K and 32K are fairly widely supported and reduce chunking-related problems.
For languages like English, there's also usually a lot of redundancy within a text, so 512 tokens might not give a very clear indication of the context. Lots of documents have similar introductions (like "#include <foo.h>\n") that make short contexts and truncation particularly harmful.
Also, "Nothing in the document past that point can ever be retrieved, and nothing anywhere told you." This is user-hostile behavior, even if they didn't want to admit to users that the auto-embedding support was poor.
Finally, the paragraph later on about truncation being "what you already have" reads like Claude talking to the developer, not like a vendor talking to users. But sure, maybe this is a good default for a database searching page titles, chat logs and Xeets?
"Chunking" has a bad history in IR going back through the history of the TREC conference. For instance you might index the whole document or index sections or paragraphs and somehow aggregate the paragraph hits up into document hits. It's one of the many things that doesn't help performance even if you think it did, and it may hurt.
The answer is "make the context window as large as you reasonably can" because you have to have enough (con)text in the window for the system to decide what the words mean and if you don't have enough of it you won't get it right.
On the 512 tokens: that's the window of the model we benchmarked with (all-MiniLM-L6-v2), not a claim about embedding models in general. The article does mention text-embedding-3-small's 8,192 window, and the same setup works with 32K models like Qwen3-Embedding. If your documents fit the window, the advice stands: keep truncate.
A bigger window makes chunking easier, not irrelevant. max_tokens defaults to the model's own limit, so with an 8K model you get fewer, larger chunks and overlap matters a lot less.
Two reasons we still chunk even when the document would fit:
1. One vector per document is a summary of the whole thing, so a short, highly relevant section gets averaged away by everything around it. One vector per chunk turns the question into "does this document contain something close to the query?", with the doc scored by its best chunk. Your #include example is exactly that case: the first N tokens of every file look alike, and what distinguishes them is further down. That's the "deep content" split in the benchmark — truncate got 55% recall@5, recursive got 83%.
2. Cost. Transformer embedding time grows superlinearly with input length, so pushing a whole 8K or 32K document through a local model on CPU costs far more than embedding it as 512-token chunks. Remote APIs bill per token either way.
That said, you're right that our numbers only show the effect against a 512 window. We should rerun the same benchmark with an 8K and a 32K model. I'd expect the gap to shrink but not disappear, and that's worth measuring rather than assuming.
On "nothing anywhere told you": agreed, silent truncation is bad behavior. That line describes what Manticore used to do (and what most embedding pipelines still do by default), not a defense of it. truncate is still the default because multi-vector output needs a different column type, so it has to be opt-in.
On "what you already have": fair, that sentence reads badly. It means "the old default, unchanged", not "good enough for you". We'll reword it.
"If your documents fit the window, the advice stands: keep truncate."
"Two reasons we still chunk even when the document would fit:"
Which advice do you stand by? Obviously, very short content doesn't need chunking, so let's consider a document that fills 75% of the input context.
When chunking, your cost overhead (per token) goes up as the number of new tokens per chunk goes down. That's an argument for longer chunks, although the averaging/smearing point argues for not going too long.
Embedding calculations are effectively prefill: on my cheapo local inference system (32 GB AMD R9700 + 8 GB AMD RX 7600), the older 8 GB card goes about 80% as fast as the bigger card for Qwen3-Embedding-4B (a bit over 19 chunks/second on my usual corpus, blog posts+comments that are mostly well under 32K tokens). So I would suggest that anyone who is limited by CPU embedding models could benefit from even a small local GPU.
For your blog post, I would suggest an explanation of the chunking modes, either in the blog post or as a hyperlink to the docs about them. "truncate" and "sentence" are fairly clear, whereas the others are not. (If "mean" just computes the mean of the embeddings, that seems like a poor choice. The arithmetic at https://www.johndcook.com/blog/2026/09/16/coffee-milk-latte/ might work for single words, but seems likely to break down at the document level. "recursive" and "fixed" are opaque, at least to me.)
If/when I index my team's documents, I will consider a content-aware chunking that fits as many sentences, paragraphs or sections as possible into each chunk, with overlap determined by the level at which the chunk finishes. Content-agnostic chunking is easier to code and more generic, but indexing should respect a document's internal structure.
Just so you know, your comment was automatically hidden (“dead”) until I vouched for it now. Same for most of your recent submissions. Actually it’s probably because of your (exclusively self-promotional) submissions that your comments and submissions get hidden.
On the bright side, lots of people will be saved from reading bad prose like "Malloc (libc) is the worst memory allocation API to use" and "Programs should avoid, if possible, allocating/deallocating memory too often". (By definition, "too often" means it can possibly be avoided, and usually that it can practically be avoided.)
The first one was offered with no explanation and no claimed better alternative. malloc() is probably the simplest interface for generic runtime allocation, and simplicity has a lot in its favor. malloc() does not provide type safety, is susceptible to external fragmentation, and makes it harder to meet the performance goals outlined in the rest of the blog post. So malloc() reflects trade-offs, but saying that the standard API "is the worst memory allocation API to use" should be supported, even if briefly, instead of simply asserted: meeting the goals I listed inflicts other drawbacks, like needing to create and manage separate heaps.
I thought my explanation for the second one was already clear: "Programs should avoid, if possible, [doing X] too often" is a truism because "too often" implies that some reduction is possible. One should leave out the ", if possible," -- although deciding what is "too often" can be challenging and sometimes a matter of taste. (Is reducing allocation frequency by 5% worth doubling the CPU usage or memory usage or code complexity? Maybe in some cases, but often not.)
So is "does [not] fix [attribute]". The opening dot list has several others, such as "X, Y, Z stay open", "the claim carries its ___", "___ is a statement about ___ and says nothing about ___", and repeating a pair of numbers followed quickly by the difference between them. (It's +0.009!)
And I have to think hard to guess what it probably meant by "Comparability is a property of the reference the results are traceable to, not of the number". I don't think "[X and Y] are different scopes by canonical bytes" even makes sense.
The whole point of the MS Access reference is that similar situations have been tropes since at least the 1990s. Bad code generated by someone who doesn't know how to program well -- whether that person is supposed to be a professional programmer but is incompetent, or has a different job -- is nothing new, and neither is having competent programmers clean it up. LLMs probably generate more of it, but can also fix a lot of it, or at least patch it up.
A year ago, LLMs were not useful for me as a programmer. Now they are: the models are better, they can use long contexts more effectively, and the harnesses are better at helping the models. Nowadays my job is mostly not programming, but LLMs let me organize and prepare tools in spare time rather than needing days or weeks of attention. I would not trust them on a 200k+ line project -- and Claude Opus 5 has issues even on 50k LOC projects -- but they absolutely can help given good direction and a narrow enough scope.
Then I don't understand at all what point you were trying to make about being miserable in our jobs. Cleaning up after bad code has always been part of the job for decades; so has balancing the creation of new technical debt against resource availability.
> The whole point of the MS Access reference is that similar situations have been tropes since at least the 1990s.
I do actually understand that.
I have done this very put the Access database on the web job myself. (FWIW I was well-paid for it and the firm I worked for earned a fortune, but this was in 1997)
That's not very close to the argument they make; the claim (right or wrong) is that large corporations often use monorepos, and those are bigger than single-project repositories like used for open source projects.
Their claim is consistent with Google's experience, but somewhat at odds with (say) that of FreeBSD, OpenBSD and NetBSD.
It's certainly not the case that no open source projects use monorepos, you are right that the BSDs do as well.
It's that those repositories are still orders of magnitude smaller than the ones that are inside of companies. Monorepos tend to be larger than polyrepos, due to the fact that you've got everything all in one repo, but that doesn't mean all monorepos must be truly large.
The real issue with the open vs closed bit is that features like "this subdirectory of the repository is only writable (or even visible!) to some people" is basically nonsense for open source, but valuable to a company monorepo. Their needs are just different.
Why "sadly"? The person writing a link should provide enough detail for readers to figure out whether they should follow the link. I rolled my eyes the first time I saw this entry on HN because it just said "Neki" and the (generic) target domain name. A two-word description got added, and that makes it interesting instead.
It's sad because I think links are the best thing about the web, and it's increasingly clear to me that many internet users either misunderstand or actively avoid them.
Why do people post screenshots of an article on social media instead of linking to it?
Sometimes ignorance, but often because the web is such a hostile experience these days - paywalls and subscribe banners and cookie warnings and slow pages.
The same thing as the last word of "That is the difference between 22 and 226 seconds, measured." Techies I know would mostly omit "measured"; the rest would show, not tell.
Drawing a strong conclusion from one shaky data point - classic human tell?
I've been reading John D. Cook for years (maybe decades? "The Endeavour" is one of my oldest bookmarks), and this post was no more written by AI than his oldest posts.
yeah, there were some posts of his that always got top hit on certain google searches in the days before stackexchange. And this sounds like typical John D Cook. All these people claim to identify some "tells" and whenever a study is done people are horrible at distinguishing AI vs non-AI prose.
reply