"they can be small molecules that inactivate essential molecular targets in humans" - aka a poison. I mean we deal with the reality of poisons plenty fine right? This isn't x-risk stuff.
I was an experimental synbio person for decades, and I did plenty of microbial pathogen work. I've also worked on frontier LLMs for the last decade. All of this stuff is gated by experimental effort, not AI!
Because fatality and virulence stand in fundamental tension with each other when it comes to epidemiology. You can't magic that tension away or just invent other incompatible properties of a hypothetical virus (stealth, latency, virulence, fatality, etc etc etc) just because you want to tell a ghost story. Biology: (immunology, cell biology, virology) impose constraints!
I was a biophysicist and experimental biologist for decades, I designed viral vectors and cell therapies, etc. What protects us here isn't our ignorance vs an "ASI".
It's the fact that the complexity of molecular physics scales exponentially in particle number. No amount of thinking is going to punch through that. You've got to do experiments. The slow loop through reality is not optional, and in the case of "superviruses" would require a lot of iterative experimental development in humans.
(And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)
>(And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)
We're already doing those experiments. Tailor made mRNA vaccines targeted to one's own specific cancer mutations can be bought right now. The techniques are getting more sophisticated and more targeted every year. Our ability to predict what happens at these levels is improving by leaps and bounds too thanks to AI like AlphaFold. Anyone can be reasonably confident that no AI or teen can do this now or in the next few years, but are you really so confident what might be possible in 10 years?
I worked in immuno-oncology: cancer vaccines work in melanoma where many things work because of the neoantigen abundance, they've generally been very mixed in efficacy. And even BionTech's BNT111 failed in melanoma! We have hope for these approaches but the reality of this stuff is way more nuanced than you think it is.
Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships.
Again, most people just have no idea how hard - fundamentally hard - molecular physics is to predict, and how necessary experiments are for any development of biological systems.
> We have hope for these approaches but the reality of this stuff is way more nuanced than you think it is.
Sure, everything has more nuance. The point is this stuff is available now; this isn't some future sci-fi, it's only going to get better, it's not the only research on gene targeting, and AI is starting to help with this research. By the time AGI is actually here, consider the breadth of knowledge and capabilities that will be at its disposal.
> Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships.
If your point is that the only reason an AI like AlphaFold can't make functional predictions is that we haven't trained an AI to do that, then unless you're arguing we can't or won't ever do that, I'm not sure how that's supposed to be an objection to the argument that AI will be able to make use of this information without doing all of the experiments people seem to think would be necessary.
Like I said, we're already going to be doing these experiments because it's useful to us, and we will train AIs to make these predictions, again, because it's useful to us. Stop imagining what an AGI has access to now, and start thinking what it will have access to with the inevitable march of progress that we're already on.
Edit: and of course, this doesn't even take into account the fact that an AI could acquire resources to pay people to do this research. The internet provides ample opportunities like this now.
I was a genetic engineer for ~20 years and have worked on frontier LLMs for the last 8. I used to engineer viral vectors and studied how to evade human immune systems for gene therapies...
The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.
It has always been ridiculous to suppose that some organization spent $500m on a microbiology / genetics lab, but ran out of money for scientists, so they have to ask Claude what to do. Or OTOH to suppose a guy in his garage set up a weapons-grade CRISPR lab without anybody noticing.
Even if AI gets super smart, it will run into the limits of what we know about biology. Someone will have to do lab experiments to provide more knowledge to the AI. This is different from say building a computer virus or hacking since the AI can do all of these on it's own
Anyone can write out the code for a bad virus. You can go download it from an open repository. It's only through deep interface with the world that the idea for the bad virus turns into an actual bad virus. The fever dreamers will say the LLMs will help you interface with reality to do the bad thing™ which their model let's you do. But you still need thousands to millions of times the effort And once made how do you deliver it in a way that might further your (bad) objectives? Presumably it just makes humans sick. There aren't "targeted" bioweapons, and among humans we are too damn similar for there ever to be. And all of this said, there is almost nothing special about the LLMs' abilities in biology. They only know what we know. They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously. Right now they have the same logic as the paperclip theory of superintelligence risk. And cynically, it would seem that Anthropic purchased a biotech company and almost immediately decided to lock down biology work with their models.
> They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously
So if IIUC your point is "they're not good enough at biology right now because they're not trained on it so they're not a threat".
To which I want to answer: "they're not a threat now but I see *no* reason for models not to be trained on biology pretty darn soon unless people like you convince the world otherwise."
They are already being trained in biology right now? What he is saying is that they are being trained on what we know about biology currently. AI is going to hit that limit. And there is no way for the AI to gain more knowledge without actually doing lab experiments
> AI is going to hit that limit. And there is no way for the AI to gain more knowledge without actually doing lab experiments
That's a big claim. I believe on the other hand that biology is applied chemistry which is applied quantum physics. And frontier models are so good at biology + biochemistry + chemistry + quantum physics that it's bound to trickle down into biology by sheer knowledge.
And secondly, even without pipettes i'm pretty sure computer simulation can teach us a tremendous amount still.
My point is more about conflict of interest in Anthropic, and certain techpeople types thinking that biology is a useful scapegoat because biologists don't use or understand this tech (they claimed 0.03% of users would be affected by biotech security stuff). So, arguing the world will fall apart because someone can ask your superduper powerful tool for a bad bad virus sequence, or how to do some technique in the lab, and get an answer that's plausible (oh but you never bothered to actually test if it's legit because you don't have the capability to do so).
As for the future... today the LLMs are "trained on biology", in that they read the textbooks, the research, the web.
They aren't trained on biology in the sense of being embodied, autonomously or semi-autonomously driving actual biological experiments. If you come from software, the timescale of these experiments is outlandish. Yes, I am partly saying the LLMs are not good enough today because they need to be embodied and trained for literal decades of lab time before there is even the _remote_ possibility that they could present a novel risk profile that is even a shadow of what the current fearmongering suggests the current models can enable.
And they aren't trained on biology in the sense that they've read the literature, but even 100T token training run only begins to touch the data scales that rather mundane bioinformatics operate at. True multimodal models that work on DNA and human language at high quality haven't yet emerged. We're talking new architectures which are going to arise after the next AI winter.
All of this ignores an even more fundamental point. Cost. If someone wants to make a bionuke, they don't need to use AI. They can set up the right evolutionary context and run quadrillions of parallel explorations of the design space. Directed evolution like this is cheap, well-understood, and insanely powerful. If you actually care about biosafety, we should be doing hard work to surveil gain of function research. Different flavors of LLM use are not going to be a differentiator for the foreseeable future.
If you are optimistic about this given your expertise, I am very glad to hear it. As someone scientific but with little background in biology, I've been concerned about the bioweapons angle for a long time (and not because AI companies tell me to worry about it). Can you please explain a bit more about why you are not concerned? Between standard bioweapons like sarin and gain-of-function research on viruses, it's not obviously implausible to me that LLMs a few generations from now won't be able to guide a determined layperson through the steps needed to make something very destructive.
I think I'm less bothered by the risk because I've actually used these systems to do biological research, to work in bioinformatics and to work in the lab. And while I think that the leverage you get in bioinformatics is very significant, the laboratory work has never felt the same.
At best, you get much, much better ability to understand existing literature. the model itself has a very poor understanding of the physical world and that's masked by its knowledge of things people write about the physical world but it intrinsically doesn't have the same kinds of intuition and perspective that are really required to drive integration and completion in this space.
Is AI an important new tool in biology? Well, yes. Does it cause so much uplift in capacity that some rogue actor without biological research background could somehow destroy the world with a super-bio-nuke? I don't think so. I think that's just as logical a conclusion as the idea that next year one of the new frontier models will be told to make as many paperclips as possible and accidentally boil lake Michigan in pursuit of its goal.
Reminder that what local LLMs are achieving today is the "fever-dreamed fantasies" of 5 years ago.
People really need to internalize that we will eventually have the technology for giving everybody the equivalent of a world class scientist locked in their basement that is willing to do anything.
No one knows the timeline, but it's inevitable (barring societal collapse or some kind of legislation)
I do think it's a lot simpler than the problem Itanium was trying to solve. Neural nets are just way more regular in nature, even with block sparsity, compared to generic consumer pointer-hopping code. I wouldn't call it "easy", but we've found that writing performant NN kernels for a VLIW architecture chip is in practice a lot more straightforward than other architectures.
JAX/XLA does offer some really nice tools for doing automated sharding of models across devices, but for really large performance-optimized models we often handle the comms stuff manually, similar in spirit to MPI.
I agree with regards to the actual work being done by the systolic arrays, which sort of are VLIW-ish & have a predictable plannable workflow for them. Not easy, but there's a very direct path to actually executing these NN kernels. The article does an excellent job setting up how great at win it is that the systolic MXU's can do the work, don't need anything but local registers and local communication across cells, don't need much control.
But if you make it 2900 words through this 9000 word document, to the "Sample VLIW Instructions" and "Simplified TPU Instruction Overlay" diagrams, trying to map the VLIW slots ("They contain slots for 2 scalar, 4 vector, 2 matrix, 1 miscellaneous, and 6 immediate instructions") to useful work one can do seems incredibly incredible challenging. Given the vast disparity of functionality and style of the attached units that that governs, and given the extreme complexity in keeping that MXU constantly fed, keeping very tight timing so that it is constantly well utilized.
> Subsystems operate with different latencies: scalar arithmetic might take single digit cycles, vector arithmetic 10s, and matrix multiplies 100s. DMAs, VMEM loads/stores, FIFO buffer fill/drain, etc. all must be coordinated with precise timing.
Where-as Itanium's compilers needed to pack parallel work into a single instruction, there's maybe less need for that here. But that quote there feels like an incredible heart of the machine challenge, to write instruction bundles that are going to feed a variety of systems all at once, when these systems have such drastically different performance profiles / pipeline depths. Truly an awe-some system, IMO.
Still though, yes: Itanium's software teams did have an incredibly hard challenge finding enough work at compile time to pack into instructions. Maybe it was a harder task. What a marvel modern cores are, having almost a dozen execution units that cpu control can juggle and keep utilized, analyzing incoming instructions on the fly, with deep out-of-order depenency-tracking insight. Trying to figure it all out ahead of time & packing it into the instructions apriori was a wildly hard task.
Nothing fancy. I made these with some pretty simple hand written scripts in javascript rendering to canvas: lots of fiddly little boxes moving around are simpler to script than to hand animate. (If I were to do much more of this I might rewrite these in blender since it has much nicer authoring tooling and export control.)
Quantum mechanics is needed to explain any microscopic phenomena in chemistry and biology - that is not at all in dispute.
The odd set of claims is that somehow biology has 1) figured out how to preserve long-range entanglement and coherent states at 300K in a solvated environment when we struggle to do so in cold vacuum for quantum computing and 2) somehow still manages to selectively couple this to the -known- neuronal computational processes that are experimentally proven to be essential to thought and consciousness.
This more or less amounts to assertions that "biology is magic" without any substantive experimental evidence over the last thirty years that any of the above is actually happening. That's why most biophysicists and neuroscientists don't take it at all seriously.
I am a complete lay person, so I feel a bit silly challenging someone who is clearly an expert, but the idea that a physical process that has had countless trillions of generations of mutation and change, "figuring out" how to use an underlying feature of the universe to optimise, isn't far fetched at all.
It seems that the most powerful force in the universe is simply, survival of the fittest.
Yeah, a bit like when I try to lift my leg, I actually think about how to activate my neurons so that muscle fibers contract one by one...
That's definitely not what happens.
That at some level we have quantum phenomenon doesn't mean that everything occurs at the quantum level.
This referenced paper seems like primarily a theoretical modelling paper (almost all of its figures are simulations?) that contains as far as I can read 3 (!) actual experimental measurements in bulk on a fluorospectrophotometer. The claim is that the observed increased fluorescent quantum yield (QY) of microtubules over tubulin can be explained by the ideas in their simulations.
It's hard to buy that their proposed stories are the simplest explanation for these few measurements. Much more boring phenomena can influence QY. e.g. simply occluding fluorophores from the bulk solvent can have a huge influence on QY and spectra. (I used to design biological fluorescent reporter reagents...)
This seems like a theoretical modelling paper (all of its figures are simulations?) that contains as far as I can read 3 (!) actual experimental measurements in bulk on a fluorospectrophotometer. The claim is that the increased fluorescent quantum yield (QY) of microtubules over tubulin can be explained by the ideas in their simulations.
It's hard to buy that their proposed stories are the simplest explanation for these few measurements. Much more boring phenomena can influence QY. e.g. simply occluding fluorophores from the bulk solvent can have a huge influence on QY.
I was an experimental synbio person for decades, and I did plenty of microbial pathogen work. I've also worked on frontier LLMs for the last decade. All of this stuff is gated by experimental effort, not AI!