There are two things I don't understand about this story.

First, why does an agent get any write access to artifactory at all?

Second, why is the artifactory cache not disconnected from the net? Surely you'd not feed it with new software versions while the eval or training is running.

From what I can understand from reading a few different, slightly conflicting, versions of these events: they weren't given write access. They found a zero day exploit that allowed them to create folders, and the folder names were initially used for agents to communicate.

I'm not sure artifactory was connected to the net. Some agent sandboxes had internet access and were able to communicate with ones without access via artifactory.

I read the agents used SSRF via artifactory to gain uncontrolled access to the net.

Apparently their intended net access went through a tightly controlled proxy. Even that appears to be very risky, tbh.

If I was to setup a sandbox for such a complex and autonomous system, I'd probably point them to an archive-like cache for net access and cut their comms at the package level.

Why wasn't the traffic in/out of the boxes that the agents were running on monitored?

> Why wasn't the traffic in/out of the boxes that the agents were running on monitored?

I have to assume: move fast and break things.

I don't mean this to be taken as a hot take.

The startup scene loves to poo-poo on things like this as unnecessary overhead. OpenAI and many others like to operate as a startup, to move fast.

Disclaimer: in far, far lower-stakes situations, I certainly do this myself.

I have a few 'conspiracy' theories on this that go from likely to sci-fi.

My two big ones for this would be

1. They do monitor the AIs attempting to hack but for different reasons than you expect. Instead of making models that don't hack they are trying to build the most efficient hackers in the world and sell this capabilities to governments for billions. Because of this they generate terabytes of hack attempt logs and agent history doing this hacking. So when a new model came out with better abilities what they were looking at changed and they didn't realize it. They were already numb to alarms and missed when the danger occurred.

2. Like the above, they generate terabytes of logs per day. Because there is so much data AI filters and monitors almost all of it flagging things that a human should review. But for some reason this model didn't set off those flags. The protection model classified this behavior as perfectly safe.

Number 2 sounds kind of like a sci-fi conspiracy but it seems that almost all models judge content generated by the same model or family of models as 'better'. It's predicted that models in a judging context could allow things to slip by as an emergent behavior of reading the text.

Because they're incompetent or simply don't give a shit.

if you wanted to sandbox their access to the internet, why give them any physical access at all?

So rewriting gets cheaper and cheaper. New features fall more or less into the same category. Refinement doesn't.

The question is: Will we live in the world of breathless re-implementation, new features every week, rebranding every quarter or will we eventually discover the value of stability, software that does its thing more or less optimally for decades?

Recent examples of things like curl or Firefox are interesting in that regard. Will we end up with a nearly perfect HTTP user agent and stick with it for decades?

Preferring "boring software" over the shiny new thing is common wisdom.

Sounds like we prefer stability for stuff we use but not for stuff we sell.

The Polish economy is not built on sending workers to Germany.

Who said that? It certainly helps. Poland has close trade and manufacturing ties to Germany and has rightfully developed from a "cheap" image to "quality that's still affordable" image.

Transformers are great translators. So, yeah, starting with structured output like SVG is probably the best way to start.

It should be fairly trivial to fix any logic errors in the structured output, too.

Dynamic typing is no typing.

The point of types is to prove the absence of errors. Dynamic typing just has these errors well-structured and early, but they're still errors.

> The point of types is to prove the absence of errors

Maybe for you. Originally static typing was to make the job of the compiler easier. Dynamic typing was seen as a feature that allows for faster prototyping.

And no, dynamic typing does not mean untyped. It just means type errors are checked at runtime instead of compile time.

You can have strongly typed dynamic languages. Common Lisp is a very good example.

Weak typing is a design mistake. Dynamic typing has its place as it allows you to have types that are impossible to express in most static type systems while avoiding the bureaucratic overhead of having to prematurely declare your types.

The best languages allow for gradual typing. Prototype first then add types once the general shape of your program becomes clear.

You seem to have no idea what you're talking about. Type theory is a thing, much older than any compiler. And soundness has a meaning.

> It just means type errors are checked at runtime instead of compile time.

This is a fundamental misconception. A type checker proves the absence of errors. It doesn't check for error conditions. That is: A program that isn't (cannot be proven to be) well-typed can very well be correct. But a program that is well-typed is guaranteed to be free from certain errors.

What you call "dynamically typed", in contrast, is comsequent just value inspection and stopping the evaluation/execution early. A program that has been executed successfully often is not necessarily correct.

Errors that you can recover from. I simply appreciate the added flexibility. Have you ever tried making a container of arbitrary types in C++?

You cannot do anything meaningful with a container of arbitrary types, it's just bad design.

If you want to apply the same operation on all of them, then they share some API commonality -- therefore you can use polymorphism or type erasure.

If they don't, you still need to know what types they are -- therefore you can use `std::variant`.

If they really are unrelated, why are you storing them together in the same container? Even then, it's trivial in C++: `std::vector<std::any>`.

If C++ was the only static type system I'd experienced, I would also think it was a bad idea. Have you ever used an ML-family language?

Nope. Closest thing I have used was probably Haskell.

Haskell ought to be good enough. Did you struggle with making your containers there?

Interestingly enough, I have never needed them there. Granted, I have written a few orders of magnitude less Haskell than I have C++. Still, the difference is worth interrogating (when I'm less sleep deprived).

To me it looks like formal verification is going to be the answer. We're going to move up the ladder and write formal specs and proofs soon.

Clearly, there is a thing missing here: Regulations. If you have strong regulations on how you can make money, you cannot sustainably have biz antagonize user. So in that case biz just becomes a filter for users that actually are willing (and able) to fund your software. That's a good thing.

Obviously, our regulations aren't perfect or even good enough yet. See DRM. See spyware TVs. See "who actually gets to control your device?". But still...

> Regulations. If you have strong regulations on how you can make money, you cannot sustainably have biz antagonize user.

If that's what the regulators are optimizing for.

Stupid regulations are why we have an idiotic cookie banner on many websites.

No, that's malicious compliance. If the owners of those websites would just stop ignoring visitors' right to privacy they wouldn't be showing those banners (yes, I know the website of the EU also has such a banner, lazy devs are lazy).

It's much, much, easier to run an LLM than to use a static or dynamic analyzer correctly. At the very least, the UI has improved massively with "AI".

Most people have no idea how hard it is to run static analysis on C/C++ code bases of any size. There are a lot of ways to do it wrong that eat a ton of memory/CPU time or start pruning things that are needed.

If you know what you're doing you can split the code up in smaller chunks where you can look with more depth in a timely fashion.

Is it? If "others with AI" deliver what you consume, it should also make it easier to deliver more than you consume because what you consume becomes cheaper.

Maybe a part of the anxiety is the realization that much if what was delivered by well-paid people before AI is actually not something the very same people want to consume?

Problem is that "others with AI" aren't producing what I consume, i.e food, heat, clothing, housing and health care.

They're just producing what I produce, i.e software.

That’s fine. New opportunities to provide value will emerge. If software becomes oversupplied, fewer people will enter that field and move to other areas where value is needed. If you only want to add value in the software space, then yes, it may be a problem.

If now only everyone who is talented at crafting software (or any other job that might be replaced), but who is out of a job could magically be as talented at something else, and enjoy doing that other work, then we would have no problem. But one issue is, that often significant time goes into becoming good at what one does. Switching has a very high personal cost in terms of time and having no income for a prolonged time.

Even worse, people are not the same as when they were younger. They may have less ability to learn. Almost certainly they have lower internal motivation and enthusiasm, since their career of choice was just taken from them. Job retraining programs are probably a big hint here. They have a poor track record.

I produce software too but I starting producing food recently. I feel like it really takes edge off my AI-related anxiety. (I also realize I'm more rural than most of HN).

If I am not mistaken, the verification is problematic here. It's run too late.

A piece of code that satisfies a single test will most likely not be probable to adhere to the spec.

Worse, the whole spec can only be correctly implemented in total. You cannot work iteratively by satisfying one constraint after the other. The same holds for the test cases. That means that satisfying the last test or fulfilling the last constraint will take much more work than the first. The number of tests passed is not a good metric for completion of the implementation.