Another way to achieve something like this is Cosmopolitan C Compiler, Actually Portable Executables. For example, Redbean. (Jart, all of your certs are bad! I can't open your websites any more!) http://justine.lol/
It's a binary that can run on Windows, Mac, Linux, and it is also a zip file. The application can read the zip file and on several systems (not Windows), it can self-modify the zip file. Including hosting SQLite inside the ZIP file.
I have been meaning to use Justine's RedBean to build a "project app" (kind of like what the OP describes, but leaning more towards something like Datasette). It would contain the data, the paper(s), and the UI for interacting with the data (and supporting scripts or R that can be run with other tools). I just like the idea of a portable single-file scientific artifact.
What happens when you ask those same frontier models to write a chess-playing program?
I feel like, this is a huge stumbling block that many people have. They'll give a model their data, and ask it questions. I vastly prefer letting the model understand the schema, and then writing functions or programs to answer those questions. I feel like I get way, way better answers. I can have it write unit tests for those functions. I can fuzz test those functions. I can look for data that doesn't fit the schema. I can process new data way faster (and with fewer tokens). I can repeatably get the same answers from the same inputs. I can check the code into a git repo and track changes to it over time. I can share the code with other people. I can review the code. I can improve the speed of the code and get the same answers. I can review the error accumulation, and improve it. I can decide how to handle anomalies, and encode those answers.
It's really neat to see what a frontier model can do itself. No doubt.
But "play chess by hand" is a frankly awful metric. It's kind of like asking someone to take a cube root of some arbitrary decimal, in their head, with no scratch paper.
To go a bit off-track based on your final sentence: my high school physics teacher would do any numerical calculation that came up first in his head, as an estimate, and only then use a calculator or write on the board. Usually the estimate was within ±10-20% of the correct value even for long combinations of numbers with a bunch of decimals. Cube roots didn't come up, but square roots did.
The point of that was to show the use of approximations and of having an idea how much a result should be, to guard against calculator typos and the like. I think that has some metaphorical relevance for the chess example.
It's a great test of cognitive abilities. There are many claims that current LLMs surpass humans in cognitive abilities so it's noteworthy that they underperform on that test.
Letting the model execute a chess program (that it wrote) would make sense if you're measuring its economic potential, but for cognition that would be cheating just like if you let a human run a chess program. The fact that the human would have a much harder time writing a useful program is irrelevant.
It isn't. Stockfish running on your laptop can beat every human being on earth easily at chess. It's not intelligent _at all_ in any sense that matters.
Well it's not a perfect test so you need a bit of care in how you use it. If you have no idea what the subject is doing, then you don't know if you're measuring cognitive ability or something else (like cheating ability, or algorithmic sophistication or whatever). But failing the test is a pretty clear sign of certain cognitive abilities being poor.
> The fact that the human would have a much harder time writing a useful program is irrelevant.
Why?
There's a box.
You give it a problem, and it comes up with a solution.
Why does it matter to you if the box is strictly a LLM, or if the LLM can write code that it executes?
Even neater if the box is self-contained with a local model. You provide electricity, and it comes up with solutions. Why does it matter if it can do chess "in its head", or if it has to use scratch paper?
The question is to what end? This is a benchmark task, because playing chess, or solving other well-understood problems is more of a party trick than it is useful.
If you let the LLM write a chess program, which it can ONLY do because there are already so many chess programs out there, then the benchmark becomes about recall of popular program source code, not chess.
Neither. As I said I want to measure cognitive abilities.
Your "ability of the box" is like "economic potential" in my previous comment. If that's what you want to measure, fine. But I want a deeper understanding: what is the thing doing, how is it solving problems? I want to get a sense of its abilities that is richer than a one-dimensional scale.
I agree that it's a fascinating to crawl inside an LLM, and also to crawl inside of a human, and try to understand the processes and limitations. Like, Phineas Gage is one of the most remarkable learning opportunities we ever had.
That said, it's really weird to me when people use (and judge) LLMs one way... and won't try using them another way.
Like, to judge their utility, I think we should be open to letting them write code, and use the code they produce.
Otherwise, it's like judging a Chromebook without an internet connection. Like, this was one of the most dishonest ads I've ever seen: https://www.youtube.com/watch?v=gDy9AUQJ3Fg
This lamp, without a working power outlet? It really doesn't do anything...
>> There are many claims that current LLMs surpass humans in cognitive abilities
Where? By whom? This is certainly not (yet) the general consensus, as I understand it. Are you taking the most optimistic / untethered comments as the strawman against which you feel the need to argue?
Why the aggressive tone and the strawman rhetoric? I never said there was a consensus. Yes I'm talking more about the "optimistic" commenters and pointing out that this chess thing is a good datum to temper their enthusiasm. What's wrong with that?
Also these claims are not completely without merit, it's just that LLMs seem to excel at specific "cognitive" tasks and it's interesting to see where they fail.
Do I care if Richard Feynman was only able to do nuclear physics with the help of an abacus?
Sure, a Spelling Bee is a fun thing to have. Little kids work so hard. They practice for hours. There's joy and heartbreak. Prized, sometimes. Notoriety. But in the real world, computer-assisted spelling is by far the norm.
Sometimes you care about the Bee, sometimes you care about the results.
This is called a benchmark. We run a calculation of Pi to evaluate a computer's performance, but we don't allow the script to download a ready-made solution. When we evaluate a runner, we don't let them use a bicycle. When we evaluate a new LLM, we don't allow it to send a request to a team of programmers, so using a chess engine for an LLM is cheating
If an LLM writes AlphaZero, and it competes with itself, and is dominant (and beats stockfish!!!), with no book positions cribbed from its learning...
The LLM has a process to beat chess.
Just like, if it doesn't inherently know how to multiply 13 * 17 without using Python to do it... I don't really care.
Maybe you do care. Maybe you want an LLM to be able to do work, only in its head.
But I kind of can't understand the desire for that limitation...
I mean, I do. But it seems ridiculously arbitrary. Like driving a car in 2nd gear and complaining that it gets terrible mileage and can't go fast enough. The Drive gear is literally right there.
The assumption is that if an LLM is incapable of playing chess—a game with a relatively small number of pieces, clear and simple rules, and perfect information—even after reading a hundred thousand books on chess, then it is fundamentally incapable of managing an army or a factory. This is because those scenarios involve more 'pieces,' incomplete and fuzzy information, and implicit rules that need to be deduced independently. It doesn't matter whether it has tools or not. It's simply that running tests with chess is cheap, whereas testing with an army or writing a browser from scratch is quite time-consuming and expensive.
Asking a model to "write a throwaway program to do X" is vastly more productive and reliable than asking "do X". Running code provides a feedback loop, the model can iteratively improve the solution instead of guess. Even if you don't read the code yourself, you have a reproducible, editable, and auditable artifact if you need it.
I'm not the one posting daily about how "AIs are going to destroy the world because of how smart they are", "humans are finished" and "we've reached super duper mega intelligence". Go see Dario and Sam about that.
>I'm pretty sure Fable could write AlphaZero
If course it does, the paper is open and dozens of open source implementations are in its training data already. It could write AlphaStockfish, or xx_chessmaster_2000_xx, it doesn't matter if it does: it's writing a solver: it's not good at playing chess. If tomorrow I tell you that I'm so fucking good at chess I can beat Magnus, and I show up with a laptop running stockfish, you're going to laugh me out of the room.
There was a feature on Amazon Web Services for a while, and I wish it was still there...
Downloader pays.
I make some content and upload it. When you want to download it, you pay Amazon the egress fees. And maybe I get to charge just a bit more, to help me with the Ingress, storage, content creation, etc.
I mean, I know that there's going to be problems with rate limiting, etc. And yes, we have those problems with LLM tokens today. But this just feels like such a useful thing that it baffles me that it doesn't exist already.
My desired usage pattern stands... Someone who publishes content shouldn't be punished for everyone else wanting to access it, and shouldn't have to resort to product placement, advertising, sponsorship, or begging to fund it.
I don't know, maybe WebTorrent should have been the answer? For upcoming, viral content?
But for the deep archives, like the Wayback Machine? I feel like I'd happily pay for egress, and a bit to support them. If it was automatic and built in...
> they don’t know what a bill of materials is for a hip replacement. Rather than identifying the actual labor, supplies, implants, overhead and other expenses tied to an individual procedure...
How often is there a malpractice case for a hip replacement?
I'm willing to bet that the costs of a malpractice case can be one, two, three, or even more orders of magnitude higher than all of those other expenses. Dwarfing any of those enumerated costs. Even cases that they win.
Of course they're looking at the statistics rather than counting beans.
Another problem is that one hip replacement is not the same as another. Differences in patients, complications arise, it's really hard to say you're going to need exactly X syringes and Y feet of gauze and Z liters of blood and which and how much of certain drugs for anesthesia, not to mention with procedures that involve prosthetics you may need multiple on hand, the first one you pick may not fit so you need to go to the second one but you have burned that first one
Finance guys think you're going to turn hospitals into walmart and make it up on ruthless efficiency and volume but it's not that kind of business. There's so much unknown going into any one situation that it makes this "transparency" hard to do. Plus, people want the best and whatever they need to survive so the "consumer" doesn't want transparency
And then you layer on top the huge amount of uncompensated costs from uninsured and denied coverage. It's why a single national payer system actually makes sense, if you can keep the fraud from running amok. We really should just pay for outcomes and spread the actual cost over the entire system
There's two different things you're talking about here. There's the cost estimate- the amount you tell people up front a surgery is likely to cost, vs the actual cost to the hospital. You don't need to guess how many syringes of X you're going to need when you've already done the surgery. At that point you just addd up all the stuff you used, price it up and that's the BOM. Cuban is saying they don't even do that - they don't track their costs on a surgery by surgey basis at all. And because they don't track any of that it's impossible to reason about it.
It's impossible to answer "How much will it actually cost to insure person X" because you don't have any of the data on what the costs will be when person X needs a given surgery, all you have is the aggregate costs of the entire system - a lot of which is misleading because things are cross-subsidized because no one is really tracking costs. It may well be that whilst every surgery is billed equally in reality obese patience are responsible for 80% of the cost. Or it may even be that the hospital is making an average loss on hip surgeries because their negotiations with the insurer drove those prices down whilst brain surgeries give a nice profit margin.
And so you can't ask "How much would it cost for the government to fund service X" because you don't know how much it costs, all you know is the aggregate money spent across all medecine.
>Cuban is saying they don't even do that - they don't track their costs on a surgery by surgey basis at all. And because they don't track any of that it's impossible to reason about it.
Then Cuban is wrong. I had a heart valve replacement surgery last year and I got a detailed to the penny to the mg of tylenol what they consumed and what they charged. It was not a big massive one line "heart surgery" on the invoice, there were dozens and dozens of line items. I think quite the contrary they know exactly what their inputs are to a surgery, it's just that is not the only cost you get billed.
>because you don't have any of the data on what the costs will be when person X needs a given surgery
This was exactly my point, and yes part of it is because of cross-subsidization, but it is also because my valve replacement almost certainly cost someone differently than the next guy having the same surgery the next day because of any number of factors
Also part of determining insurance costs is that you probably don't know which surgeries someone is going to need at the time the premiums are being set. Even if you knew perfectly how much every surgery in the system costs, you don't know how many of those surgeries you are going to have to pay for
And what if some of the components of X expire? That means you have to pay to keep them on hand.
And the distribution of how often you do X is absolutely not predictable. Sometimes it's months between them. Sometimes it's 10 in a day.
If you had to be prepared to serve 20 hamburgers, any given afternoon, with no warning, how much ground beef would you waste, over a year?
Counting how much ground beef you did use does actually provide some information. I'm not discounting that. But it absolutely does not tell the whole story.
There is a financial product known as malpractice insurance in which the outsized cost of a malpractice case is spread out over a risk pool of people with similar risk exposure.
In any case, this cost is not directly part of the "hospital's" cost of service since the surgeon is most often a semi-independent contractor who bills the patient "professional charges" that are separate from the hospital's "technical charges."
Not sure what you are trying to say? Yes past performance of investments is no guarantee of future results. It’s a pretty good gauge among other factors for measuring risk in insurance though. Imperfect yes but pretty darn good.
Going back to the original comment, your not wrong that hospitals are thinking about malpractice in cost (or should be) my point was that it should be fairly measurable in the base case and tail cases should be few and covered by other riders or self insurance.
Either assign an equal portion of your insurance to all procedures, or use the amortization techniques insurance uses to assign a malpractice risk to the procedure.
Context is important. 100% tariffs on Canada is stupid, 100% tariffs on China because we don’t want any industry dependent on a major military adversary is something worth discussing even if the answer is to not do that in the end
It would certainly help if people wouldn't believe flat-out lies.
Like, "other countries pay the tariffs."
No. They don't.
How can we have nuanced conversation about the trade-offs, if the leader of one political party blatantly lies about a fundamental detail of how tariffs work?
600% reduction in the price of drugs?
We can't even start a discussion with people who believe this. At all. They reject any information, even their own knowledge of how numbers work.
I have never met anyone who thought other countries paid the tariffs. Im sure they exist but this seems more like something retractors say to make themselves feel smarter than people in support of tariffs. People who care at all about tariffs tend to know how they work.
The president says a lot of really stupid things and those stupid things are highlighted, but that’s not indicative of the average and one of the main talking points about tariffs is that they increase costs in order to incentivize local suppliers
Merriam-Webster: "The meaning of TARIFF is a schedule of duties imposed by a government on imported or in some countries exported goods."
Apparently the dictionary is "woke."
Wikipedia: "A tariff, or import tax, is a duty imposed by a national government, customs territory, or supranational union on imports of goods and is paid by the importer."
Oxford Economics: "Tariffs are taxes imposed by a government on goods and services imported from other countries."
It's a pattern that occurs in commercial software yes, but Open Source has its own version of building mirror ecosystems. Collaborating to bleeding edge issues on robust infrastructure is very hard, so junior engineers that want to collaborate in open source (often for reputation building) can do so in these mirror ecosystems, it's an added bonus if they believe that they are contributing to the bleeding edge instead of rebuilding an enantiomer tech stack.
He worked at Google for 12+ years. He created memcached, WebSub (PubSubHubbub), OpenID, and Perkeep (Camlistore). He's a late-stage co-founder of Tailscale.
If he's a junior engineer, then I have no idea what a senior engineer is.
The CLI doesn't let you do as much as the go library does.
reply