I am begging you on my knees to please stop posting this cringe.
The model is just out. It could be good, great even, I don't know. But I do know that this index has Opus 5, one of the worst releases of 26, ahead of Astra. What information are we supposed to deduce from number having gone up?
What's more valuable than a good benchmark? IMO a benchmark that has been run against very many competitors and versions. Collecting data has something going for it, and it's up to the readers to interpret and make the best use out of it.
This index doesn't have "Astra" and "Opus 5". Every entry with corresponding data is a `(model, reasoning)` tuple.
So I'm unclear what you're actually saying and wondering if you've missed that. Are you saying that at every reasoning level it says Opus 5 beats Astra? I just compared Opus 5 high to Astra high and it has Astra as generally better than Opus.
I don't try to say that the parent commenter is right in any way, but the two models' "high" settings probably doesn't mean the same thing. So probably comparing only them is not useful.
You forgot to include whatever you're proposing instead.
"Trust me bro, Astra is better" isn't perhaps as useful as you seem to believe. I'm not even saying it is right or wrong, just that my opinion on this topic is still just one additional subjective data-point.
Only thing I wish with these benchmarks is that they would run repeat tests every couple of months. Then re-rank based on that too. We've seen a lot of performance fall-off after a couple of weeks with new releases.
Yeah unfortunately the benchmarks are usually provided by the company themselves, unquantized, thinking set to extra-extra-ultra-high, best of 10 runs, etc etc. It's hard to know how that's going to map to real world users.
That it's better in specific ways? What difference does it make when it came out? The benchmark results are not going to change unless they're messing with the model.
Yes, but crucially, in ways that are increasingly decoupled from any practical pattern of usage, considering that I wouldn't see how you can argue that Astra is worse than Opus 5.
One man's modus ponens is another's modus tollens I guess.
It's endlessly fascinating how culture, religion and even language influence stylistic choices, even though Chess is a purely abstract game.
The obvious ones: in Western sets the king has a cross on top, and Islamic traditions only use abstract shapes. But I wonder if the "usual" designs aren't at least somewhat influenced by the English language specifically. Bishops have a stylized miter and Rooks are a tower with crenellations, but e.g. in Russian Bishop is "elephant" and Rook is "boat". Even in many other Western European languages "Bishop" as in the piece is not the same as bishop as in the religious office, e.g. "fou" in French, "alfil" in Spanish, "Laufer" in German.
The reason I hate chess is exactly the bishop symbol. I cannot explain it but I just can stand it. The idea on my mind of a bishop in war. I wish future generations will go back to the elephant as symbol internationally.
What feels more off to me is why does the king have a cross? And why does a religion use the murder weapon of its messiah as its main logo? What if He was killed by a rock or a rifle or a guillotine? What would the king be carrying on his head?
1. Because many ancient monarchs claimed they had divine backing as the basis of their right to rule. A ruler wearing religious symbols strengthens that association. Chess kings from non-christian countries don't have crosses but go with other symbolism.
2. Because the most famous miracle performed by Jesus is that the crucification didn't stick: he rose from the dead after a few days and ascended to heaven not much later. So the cross is not so much a murder weapon as a callback to the central promise of christianity: that believing will grant you eternal life in Heaven after you die. It should be noted that early christians didn't really use the cross as a symbol btw, that came much later.
> (This approximation should be familiar to many from an algorithmics class.)
You need both sides though :)
What makes it interesting for estimating algorithmic complexity is that \log{n!} \in \Theta(n \log n). One side is obvious as you note, the other less so, but there's a famous trick to do both at once:
The new procedure targets inappropriate behavior by parents and spectators directed at children, coaching staff members, referees, match officials, or other parents and spectators. This includes, but is not limited to:
Verbal abuse (insults or profanity), as well as physical assault or shoving of one’s own children;
Verbal abuse, physical assault, or shoving of staff members of one’s own team;
Verbal abuse, physical assault, or shoving of players or staff members of the opposing team;
Continuous challenging, verbal abuse, physical assault, or shoving of referees and match officials;
Verbal or physical altercations with other parents or spectators.
<<<<<<
If that's the situation in, and it can't be stressed enough, a children soccer game, then yes, abandoning the match is probably the right choice. I assume they are only defining a formal procedure for practices that already exist.
>If that's the situation in, and it can't be stressed enough, a children soccer game
It's not extremely common in my area, but it is more common than you would think. Parents treat their children like lottery tickets when it comes to athletics. And God help you if you're the unfortunate umpire at a travel league game that makes a questionable call.
The youth sports industry is fucking gross and nobody seems to notice.
Source: we will not let our children participate in travel leagues and they do not want to participate in them either, for this specific reason.
I umpired baseball in my early teens and even 30 years ago it was bad.
Grown men getting into the face of 14 years old umpires over calls, at games for kids barely out of tee-ball. It just doesn’t matter that much.
I remember a coach who would flip over 20 gallon coolers and send ice and drinks flying in the dugout.. in front of his team of 10 years old umpires olds.
In Texas, there's an association, TASO, that is in charge of referees for all sports. There is no requirement about having experience in officiating at all. You attend a few meetings, pay the fees, and start officiating. The crazy thing is that once a TASO member, you can move from sport to sport. It's easy to see officials that are definitely out of their depth. As a player, it is insulting that they can't play their sport because the one in charge of the match is not very good. As parent, it's aggravating because their kid and teammates are getting boned. After personally experiencing the literal "inside baseball" of Texas high school officiating, they bring some of the flak on their officials in their attitude on who can be an official. I knew people that just slid from one sport to the next based on season as a source of a fairly decent income. Volleyball refs just sit in a chair, so physical fitness doesn't really limit how good of an official they can be. Soccer refs on the other hand...
That's the association's problem, not the referees. When I refereed basketball in Australia, referee grading was done according to FIBA standards and you had to be evaluated before any promotion in your referee grade and that could be (at the lowest levels) 3 games on 3 occasions through to 10 games or more. And when we got to a certain (albeit relatively high) level, we'd have every game recorded and reviewed.
No disagreement, which is why I said that the association is to blame. If a ref isn't good enough yet they still continue to be assigned games, then the org allowing that to happen is at fault. The immediate response is that they don't have enough officials to to start sitting officials based on performance/ability. That's just further indictment to me, but "it's just a kid's game" blah blah.
Baseball in particular has a problem with anger is the arguments. Like there is no argument without a hostile tone. Players/managers never seem to merely appeal a call -- they jump straight to harsh yelling.
Compare it to soccer where there is also a lot of whininess in the complaining about calls.
It's become accepted that vulgar, angry, disagreement is acceptable until it goes "too far".
Tantrums are just socially accepted in baseball, and some other sports.
Probably being born from public schools, and teachers being the referees, they don't have to take any nonsense from players and can take the ball away from you, lose you ground you have gained, or card you if are abusive to the referee.
Abusive parents on the sidelines can result in sanctions on clubs.
Even if you think the referee made a bad call, it is never acceptable to be abusive to match officials.
It blows my mind the things I see in football they allow people to get away with.
I love how in rugby if two guys are going a little two hard at each other the referee will call them over along with their captains, and address the captains while these gigantic players stand there like children.
One other offshoot of rugby officiating is they will admit wrongs and they'll say things like "I didn't see that." Compare that to the NFL where the refs make a bad call and the league will back them up to the point of being ridiculous by reinterpreting the rules to make the officials right (see "not a catch").
Absolutely - I used to referee basketball in Australia, including at a high level.
Extremely high level junior, was generally not an issue. As was true local league.
It was the intermediates, the travel leagues. Where "your kid is good but is still not going to be a pro" (the NBA sends a letter to all new D1 college players congratulating them on the effort they've put in to get this far - and to focus on their education, as the NBA drafts <1% of D1 college players, let alone anybody else). Those were the ones where the parents were convinced that their kid was going to be a pro and the only thing holding him back was "that fucking referee".
We had the power to eject those parents from the venue. Which was usually a conversation that went something like this:
Ref: You need to leave the venue until the end of the game.
Parent: It's my fucking kid's game and I'm not leaving and you can't fucking make me.
Ref: You're right, I can't remove you from the venue. I can however forfeit the game on your child's behalf.
Kid: Daaaad. [usually weary as this tends not to happen just the once]. Go and wait in the car. Pleaseeeee.
Parent: I will go and wait in the car, and we're going to wait and when you come out, you and I are going to fucking get into it, ref.
Cue security, if not police, walking refs to their cars.
I spent several years as a soccer ref including high school matches. It never failed that every time I hit my whistle, the side of the stadium opposite of the direction of me signaling the foul would always erupt in jeers. You could always tell the helicopter parents complaining about non-calls not appreciating that soccer is a contact sport. There was only one game where the officiating crew was escorted by police, but I really felt that was over blowing the situation. There was the one game were we had to temporarily suspend the match because of gunfire in the area not aimed at the pitch, but uncomfortably close.
The funny thing is, I became a ref because I was giving the ref of one of the games I was on the touchline coaching wasn't abusive, but definitely a constant barrage of second guessing. At the end of the match, one of the association members suggest that if I didn't like the level of refereeing that I should become a ref myself. I accepted the Pepsi challenge that rest of history. My favorite experience was a coach that was getting a bit grating. I remembered a line an old timer mentioned that I decided to give a try, "hey coach! you got an assistant? you better get 'em warmed up!"
As a coach, the absolute worst part of the 10 year experience absolutely was the parents. Luckily, I never had abusive parts. It was always over excited cheering for the dumbest things from people that had no clue about tactics and thinking that blindly booting the ball up the pitch was awesome. We had no player with the speed or skill of going over the top route one, so it was just a dumb play. Usually, by the end of the season some of the other parents could usually get them calmed down. It also helped to remind some of the over the top parents that this was recreational play, and everyone got the same t-shirt at the end of the season. Nobody was getting scholarships. It was basically bragging rights for the kids back at school.
As teens my brother and I both refed youth soccer. The parents were the absolute worst part of the experience. One of them actually followed my brother back to his car after a game to yell at him more. It was so bad that the ref association took the guy to what I've always called "soccer court" (not sure what it actually was) to have him banned from coming back for the rest of the season. That was a U-12 girls rec game by the way.
That's the perfect age for shitty parents as that's the age where the "good" players have left rec and moved to club teams so that the ones that are left are the ones with bent out of shape parents for their kid not making a club team.
Not all parents are assholes, and as a solo coach, there's no way I could have survived without some of the parent's support with various aspects of managing a team. By the third or fourth season, the parents that had started with me would help wrangle some of the other parents.
> By the third or fourth season, the parents that had started with me would help wrangle some of the other parents
This is where we were with one of my kids' teams. Once they got older and split into the elite-national/regional club level (about U12), the parents were not healthy. Over time, the parents started to realize that their kid wasn't likely to play D1 college soccer, so there wasn't a point in being "those parents".
It also helped that a few of us that either coached, played, or ref'd would "help guide" the other parents. It is now to the point where I wouldn't want my kid to change teams because our parents are so awesome.
Humans not dying would be an excellent feature for the future to have. Not a hard requirement though, we'll see if there are some story points left in the sprint.
I've got bad news for you on the whole humans dying thing. Seems like actually literally everyone is going to die. Turns out that this has actually been happening for a really long time - possibly since day one - and we've got no idea how to fix it.
I keep reading about AI killing all humans but I have yet to see a practical scenario where that happens. Somehow everyone is scared of that but nobody can explain exactly how that would happen?
A lot of people have explained how that would happen, in excruciating detail. The reason why this wasn't a major part of the discourse is that most people (including myself) did not believe jumps in capabilities like those we have seen in the last few years were likely.
For a superintelligent AI trained with methods anywhere close to what we are doing now, killing all humans is the default outcome. I actually think researchers are downplaying the risks to make their arguments more normie-friendly.
To be fair Codex and Claude Code don't have a lot of secret sauce either. This is not to say that the harness doesn't matter at all or that you can't do any better, but whenever they benchmark it always turns out that you don't lose that much with even a very minimal agent. If a different harness makes better sense for your workflow I don't see why you wouldn't switch (or make your own if you have the resources to do so).
I'm also sticking to first-party agents, but that's more because I feel no compelling reason to switch and CLI agents fit well with my spartan technology choices. YMMV.
> The ARC-AGI-3 scorecard is extremely misleading (...)
True.
> Regardless, the result is still valid (...)
If you think the game is rigged, the virtuous thing to do is to point that out and refuse to partecipate; making up your own rules is something I just don't understand, especially since the rule-abiding result would still have been SOTA.
> in the sense of passing the most famous benchmark designed specifically to measure AGI progress
The benchmark does not measure AGI progress or progress towards superhuman intelligence, as explicitly stated by the creators.
On the AGI question: surely you realize this depends on how we define the term? For example, one of the definitions OpenAI originally gave is "capable of doing most economically valuable work", which almost certainly Astra, as impressive as it is, would fall short of. I'm not saying it's a good definition, but as far as I'm concerned it's as good as any. More importantly, I don't think that it would change much if we said yes or no. I'm only bothering to take a position if it amounts to something.
This can feel as "moving the goalposts", and to some extent it is, but if done honestly "moving the goalposts" is how you make progress. Had you asked me 10 years ago I would have said that anything that could hold a conversation like GPT-4 could would probably have been wildly superhuman at almost everything. It shouldn't be hard to find ways GPT-4 was lacking, though. We see new things, we reassess and try again: that's how it's supposed to work.
The model is just out. It could be good, great even, I don't know. But I do know that this index has Opus 5, one of the worst releases of 26, ahead of Astra. What information are we supposed to deduce from number having gone up?
reply