This is a fully local voice AI pet (ASR + LLM + TTS) that runs entirely on-device on consumer hardware.
It uses:
- Optimized qwen-3-asr which has been converted for voice streaming (will open source this).
- Quantized gemma4 models with custom megakernel inferrence + speculative decoding as the brain (I have a finetuned version for specific scenarios but gemma empirically worked better as generalzed)
- Threejs for the rendering of the terrarium/jar and the 2 sample game demos that come with it
The thing I wanted to test was whether an AI character could feel like something other than a chatbot.
The dog sleeps, gets bored, asks for things. He wanders around, reacts when messed with, etc.
Or let him "out of the jar" to play games with you. Instead of just talking to an LLM about a world, you can say things like “go over there,” "pick that up," “follow me,” or “check what’s behind that,” etc, and the character can actually act on it.
The download is admittedly pretty chunky 9GB because all the models are bundled with it, but once completed it never contacts or sends any data outside again.
I think the doom demo uses a text representation of the world and it's basically, "projectile coming your way" -> "Strafe". "Enemy ahead" -> "shoot. So it works well when spawned in a room of enemies (as we see in the video).
If self driving is red means stop, green means go, and stay in your lane - then it would work great, but having to actually think and test which maneuver is optimal for a given situation while weighting safety, road rules, random unexpected actions and getting to your destination, I think it's a much bigger problem. A bigger model specifically trained on that maybe would do great, but then the output is not the constraint anymore.
But I haven't tried the model, so I 'm just ballparking and could be very wrong.
But how did they know about it? Did the agents independently discover it?
Did the model know about it because this vuln was public somewhere and systems weren't patched yet?
Same for smartphones. There was a time of unlimited hype and secrery around new iphones. Who remembers the story of an iphone 5 prototype left a bar. Journalists were going crazy, people were signing petitions for Apple to not hunt down but instead forgive the employee that made such a grave mistake. People were offering millions to buy the prototype so they can brag they got the new phone 2 weeks before everyone else did.
Or who remembers the dancing disease of 1518, were people would stop what they are doing and start randomly doing the same dance. The lords? Out of their minds. The priests? Terrified the devil had taken hold of the flock!
I have come to believe that it was probably some tik-tok like hype trend of doing a fortnite dance while waiting in line for bread and communion. And the energy back then, like now, was off the charts.
Hype and memetic trend seeking encoded deep in human psyche.
> Who remembers the story of an iphone 5 prototype left a bar.
That was the iPhone 4 actually which was special for having the first “retina” screen. It was in a case to make it look like a 3GS to be used for field testing. The journalists that got their hands on it and published about it before the announcement could not turn it on past I think the Apple logo and a message saying to return it to Apple (or maybe it was just completely off, my memory is fuzzy), but were able to confirm the pixel density via microscope and I remember it blowing everyone’s minds at the time.
> people were signing petitions for Apple to not hunt down but instead forgive the employee that made such a grave mistake
FWIW I asked about it when I worked at Apple and he was indeed not fired and I think may have even still been working there when I was there around 10 years ago (though don’t quote me on that last part, he may have left already and I’m misremembering).
He was apparently not fired or even really reprimanded since it was a genuine accident and he wasn’t the one that sold it to the press, but that did start a slew of new policies around accounting for work devices.
I had dev fused phones for open carry outside of the office while I worked there but had to register when I got them and when they were returned, which apparently didn’t used to be tracked so tightly until that incident according to my coworkers who had been there longer.
This is part of the reason I wanted to announce it even before I feel ready to ship... The frontier models have such crazy velocity (admirable) and moving so fast that I didn't want them to make an announcement and suddenly trivialize all the work I 've put into this, overnight.
The fun technical challenges (that can also act as any sort of weak moat) are being taken away one by one, on an almost weekly cadence now! :)
So, this is what I have been thinking (and part of the reason why I want to "cook" it more before shipping). Since I have been building it for so long, I am not sure how much "overfit" it is on my data, the way I speak and the phrases I 'm using.
And of course the opposite too, how much the dog trained me to speak to it a certain way to maximize outcome success.
But then I thought, when people play games they are not using highly sophisticated vocabulary and there is probably lots of repetition since they are always under some form of multi-tasking stress (playing and replying/speaking). So maybe... maybe, the system can adjust itself. Use a big LLM offline to say "user said X, we did Y - was that good?" - then retrain itself.
The decomposer is basically a bunch of old-school embeddings/classifiers stitched together, it can train super fast and doesn't need tons of data. Could the thing calibrate itself to the user? Does it even need to? (because as I said I 'm a datapoint of 1 and I am not ready for the potentially huge stream of bug reports when I ship (add some perfectionism to the mix and you get the idea)).
When watching the vids, I remembered talking (mostly text chat) in MMORPG's 20 years ago. Even when we started using Skype, etc., chatting was frequently short, slang, and quick info. Like, "pulling", "OTM", "BRB", "OOM". Or even just "let's go", and so on.
But with some practice sessions, I suspect this system would pretty quickly gravitate to using any players slang speech. Am I wrong?
Yes, the dog has a few levers including a "boredom" counter that will trigger it to do things on its own (like roam around, eat something, chase something (eg a rabbit) if there's one around etc). All connected with its current state (Eg hungry = will probably prioritize eating but not guaranteed). Other things are mapped to game events. Deterministic in nature but its underlying values can evolve over time (both its default "homeostasis value" but also the curve of which it increases). You can think of it as letting an llm design a behavior graph, and then playing that behavior graph until it changes again.
Talking to it alleviates pressure so that mechanism doesn't fire much, but in the void-mode video on the website, at the very end, it chooses to chase down an elk on its own much to my surprise.
After play testing it I had to tone it down quite a bit as it caused continuity issues...
In one scenario the dog was hungry so he went and ate something that was owned (counts as a crime attributed to the player) so we ended up in jail. Being transported to jail fired a new location event and a switch from "exterior->interior" which the dog responded to by having the llm author something along the lines of "It's nice to be finally indoors, it was getting cold outside". Which made for a very funny moment but a frustrating gaming experience.
I will be open sourcing it soon :) my code is a bit dirty (the whole system is 3 pieces. The game adapter / a websocket bridge between game and brain / the brain itself) and it runs on 2 different (local) machines currently.
Ale is what makes this work locally, I felt a little conscious about it as I am not sure if it is a novel approach or somebody comes out and claims I rediscovered BERT or something (though ale runs at 1/10 the cost of BERT).
This is a fully local voice AI pet (ASR + LLM + TTS) that runs entirely on-device on consumer hardware.
It uses:
- Optimized qwen-3-asr which has been converted for voice streaming (will open source this).
- Quantized gemma4 models with custom megakernel inferrence + speculative decoding as the brain (I have a finetuned version for specific scenarios but gemma empirically worked better as generalzed)
- Pocket-tts with added vector embeddings for emotion, optimized for CPU inf (https://pantel.is/projects/pocket-tts-raven/?b=1).
- Threejs for the rendering of the terrarium/jar and the 2 sample game demos that come with it
The thing I wanted to test was whether an AI character could feel like something other than a chatbot.
The dog sleeps, gets bored, asks for things. He wanders around, reacts when messed with, etc.
Or let him "out of the jar" to play games with you. Instead of just talking to an LLM about a world, you can say things like “go over there,” "pick that up," “follow me,” or “check what’s behind that,” etc, and the character can actually act on it.
The download is admittedly pretty chunky 9GB because all the models are bundled with it, but once completed it never contacts or sends any data outside again.
reply