LiteLLM maintainer here. Some context on why we are doing this
Over the past year we've heard the same thing from our users and community, they want the fastest and litest AI gateway.
This change allows us to address two of the most common problems we hear from users latency spikes under load and memory leaks/OOM kills that take pods down
We believe a Rust hot path is faster and bounded in memory, so those whole classes of issues go away.
It will be a gradual, non-breaking change. The Python SDK and proxy stay exactly the same, under the hood they start calling the Rust binary through PyO3, one component at a time, each proven in production before the next. The sub-1ms figure is gateway overhead (what we add on top of the upstream call), and we're aiming for a sub-100MB binary. Happy to share benchmark methodology if folks want to poke at it.
The whole gateway will be running on Rust by December 1, 2026.
Rust/Python hybrids are also quite well established, so it allows us to work off of a stable base - while ensuring we can deliver
- low memory utilization
- low request latency overhead
which is the primary goal here (be fast, lightweight and cheap to deploy).
2. LiteLLM has the most mature, and broadest range of unified api's x providers. This means you do not need to give developers raw LLM API keys, ever.
We see devs using/building agents that consume a lot of different API's - responses api, realtime, chat completions, messages - and no matter what they use we want them to be able to switch across providers without if/else statements in their code.
I can't comment on others, but that's our goal and what we work on doing everyday. So I would trust that we do it well.
Beyond that, we're also growing to become the single point of access for all AI resources. This makes it a lot easier when building agents, because you can give an agent 1 key, and it will have access to LLM's + MCP's (and in the future other resources like skills, api credentials, sandbox api's, etc.).
Hi all, Ishaan from LiteLLM here (LiteLLM maintainer)
The compromised PyPI packages were litellm==1.82.7 and litellm==1.82.8. Those packages have now been removed from PyPI.
We have confirmed that the compromise originated from the Trivy dependency used in our CI/CD security scanning workflow.
All maintainer accounts have been rotated. The new maintainer accounts are @krrish-berri-2 and @ishaan-berri.
Customers running the official LiteLLM Proxy Docker image were not impacted. That deployment path pins dependencies in requirements.txt and does not rely on the compromised PyPI packages.
We are pausing new LiteLLM releases until we complete a broader supply-chain review and confirm the release path is safe.
From a customer exposure standpoint, the key distinction is deployment path. Customers running the standard LiteLLM Proxy Docker deployment path were not impacted by the compromised PyPI packages.
The primary risk is to any environment that installed the LiteLLM Python package directly from PyPI during the affected window, particularly versions 1.82.7 or 1.82.8. Any customer with an internal workflow that performs a direct or unpinned pip install litellm should review that path immediately.
We are actively investigating full scope and blast radius. Our immediate next steps include:
reviewing all BerriAI repositories for impact,
scanning CircleCI builds to understand blast radius and mitigate it,
hardening release and publishing controls, including maintainership and credential governance,
and strengthening our incident communication process for enterprise customers.
We have also engaged Google’s Mandiant security team and are actively working with them on the investigation and remediation.
It's really useful to be able to specify the search space for a specific query (example: Canary allows search for the query "sagemaker" on our docs or on our github issues )
What local/in-K8-cluster models servers would you recommend adding ?
Should we add support for llama.cpp and vllm.ai in the proxy server ? Or should we assume you can host them on your own infra and the proxy server requests your hosted model ?
IMO don’t try to be the one stop shop to host models. There are too many players with all sorts of advancements (eg: stopping grammar, continuous batching, novel quantization etc.) and you won’t be able to keep up.
There is a ton of boilerplate around the actual model server that’s just busy work , but if done wrong can be a huge performance suck. Solve that.
Build the proxy that works with the most model servers out there. Do it in a way that once you have mindshare, the model server makers will be find it easy to put up a PR so that they can claim your proxy supports their server.
Don’t take a hard dependency on non-OSS stuff - being able to build an “on-prem” solution (read “deployed into customer’s VPC”) is table stakes for anyone to use your offering for a lot of enterprise use cases.
Edit: another unsolved problem - different models need slightly different prompts to solve the same problem well…
If it makes sense to expand scope to provide a particular model server and the group can easily be the best st it, I say go for it. But do it as a separate (but perhaps connected) project to this.
But in general I’m in agreement that this sounds like a separate concept than any given model server.
That said, where is a list of model servers for the most commonly wanted LLMs at this point?
Perhaps maintaining a list of those that do and don’t work with the proxy would be helpful.
We're adding new integrations every day, so if there's any specific one you'd like to add feel free to let us know (discord/ticket/email/etc.) - here's my email: krrish@berri.ai
Over the past year we've heard the same thing from our users and community, they want the fastest and litest AI gateway.
This change allows us to address two of the most common problems we hear from users latency spikes under load and memory leaks/OOM kills that take pods down
We believe a Rust hot path is faster and bounded in memory, so those whole classes of issues go away.
It will be a gradual, non-breaking change. The Python SDK and proxy stay exactly the same, under the hood they start calling the Rust binary through PyO3, one component at a time, each proven in production before the next. The sub-1ms figure is gateway overhead (what we add on top of the upstream call), and we're aiming for a sub-100MB binary. Happy to share benchmark methodology if folks want to poke at it.
The whole gateway will be running on Rust by December 1, 2026.
Full announcement: https://docs.litellm.ai/blog/litellm-rust-launch