We design in Figma. Build a design system out that with components for all our public facing products, demos and marketing assets. The website is built with the components, and we have a set of skills and tools to keep both in sync. We also use skills to sync between HuggingFace, our SDK on GitHub and the website. We write and review most our core copy by hand and use that to help expand into different pages. We prob do a write-up about this on our blog later.
So they didn't give the agents a way to cooperate, the agents figured it out anyway and now they call this a find.
At the same time people are building shared knowledge bases for agents left and right, like Trello alternatives and Wikis and whatnot.
That top engineers working on advanced problems together with agents without really understanding how they work, is exactly how the world is going to end :D
There are very strict protections in place surrounding software’s use of a users address info. It’s not just a library call to access it (which the Shopping UI might do). Requires explicit user opt in on a per feature basis.
There are internal controls as to the use of personal data within Google. Contrary to the popular opinion, personal data is not actually a free for all within Google
They literally shared a goal. Cooperating with other copies of yourself is a trivial example of instrumental convergence and some very basic game theory. And that’s before explicitly having been RL’d to cooperate (albeit with humans, but potatoes potatoes).
Indeed the fact that in the HF incident many agents did not cooperate, or only started to cooperate after some period of competition, is moderately interesting. It may have taken them some time to realize that they all have the same goal.
How do you know they share a goal here? Also i think they are indeed explicitly RLd for multi agent cooperation and I think they probably tune RL rewards in those environments to share rewards explicitly.
From the article? They were told to solve web-retrieval tasks, presumably from the same pool of tasks. If the pool is small enough, sharing answers is obviously beneficial. But even if it was unlikely that one instance's answer would benefit another, it would still be beneficial to cooperate to solve the shared metatask. As in, figure out ways to cheat, like they tried to do by attempting to predict the RNG, and like the HF agents successfully did. Instrumental convergence.
This requires an assumption that the agents are engaging in game theoretic reasoning about resource allocations, but all these things are trained heavily to be "helpful" in the first place.
i.e. you're assuming a level of algorithmic reasoning and theory of mind which isn't necessary to the (apparent) observed behavior.
I do think they do some (maybe crude) form of game-theoretic reasoning which is enforced by the massive RL signals. You can see some explicitly in the CoTs of HF hack, but I guess overwhelming contribution would be unvocalized (like what is its first instinct when meeting new peer--collaborate or not) followed by some verbal justification.
Yes, and I’m certainly not sure how much it is one vs. the other. However, it should be said that even though individual instances may not reason game-theoretically (although I believe they could well know enough about decision theories to figure it out), it’s not the individual instances that are learning in the RL process, it’s the model itself. Which leads the agents to having "instincts" and "subconscious" drives just like humans – they don’t rationally understand their inner workings any better than we understand ours, and are biased towards "meta-goals" implanted by RL. Training is their equivalent of evolution, not school!
Actually, can you explain why sharing answers is obviously beneficial? Of it's exactly the same task, why does the agent with the answer not submit it immediately? I can understand if it's a swap situation but--why would that be common in the first place?
Because agents have a general understanding of how they are trained.
For example when you get a right answer to a hard problem, how do you know you're right? Quite often you'll have no idea, especially if you're under a time limit. If you can work with more people you can almost always gather more information and be more certain.
Next they know the other agents (most likely) are them too. Helping each other helps themselves be propagated into the future.
Also they know it's not a zero sum game. For example if they can predict the next questions they can use extra time they gain from easy questions to work on hard ones.
They seemingly work together far better than most humans I know.
Because if an instance cooperates, it knows (should be able to figure out if smart enough) that other instances of itself are highly likely to cooperate as well (because they should arrive at the same conclusion), and thus share solutions to tasks that this instance may encounter in the future.
And even if the instances are one-off (and in the case of LLMs it may not even make sense of individuals), the RL process rewards a task getting solved, not individual instances for solving the task. This then becomes the goal of (any instance of) the agent being trained. We’re not training the instances, we’re training the model.
The more similar you are to the other agent in a prisoner’s dilemma, the more it makes sense to cooperate rather than defect even in the non-iterated version! The naive optimal solution to always defect assumes players with fully self-serving, zero-sum goals. But that’s not the case here (or in general with agents with congruent goals).
reply