How We Built a Three-Agent AI Development Team — and Why It Works (Part 2)

The Config File That Started It

At some point during the development of our internal meeting AI tool, `llm_config.json` looked like this:

```json 
"deepseek": { "url": "http://***.***.**.**/v1/chat/completions", 
              "model": "deepseek-ai/deepseek-r1-distill-llama-70b" }, 
"mistral":  { "url": "http://***.***.**.**/v1/chat/completions", 
              "model": "mistral/mistral-small-24b-instruct-2501" }, 
"minimax":  { "url": "http://***.***.**.**/v1/chat/completions", 
              "model": "MiniMax-M2.7"} 
``` 

Three different URL. Three different model name formats — one with a namespace prefix, one without, one with a proprietary tag. Authentication handled differently per provider. The application code had to know about all of it.

This is the problem LiteLLM solved. After the change, the same config became:

```json 
"deepseek": { "url": "http://***.***.**.**:9400/v1/chat/completions", 
              "model": "deepseek-r1-distill-llama-70b", "api_key": "" }, 
"mistral":  { "url": "http://***.***.**.**:9400/v1/chat/completions", 
              "model": "mistral-small-24b", "api_key": "" }, 
"minimax":  { "url": "http://***.***.**.**:9400/v1/chat/completions", 
              "model": "minimax-m2.7", "api_key": "" } 
```

One port. Consistent model names. Unified authentication. The routing is LiteLLM’s problem now, not the application’s.

What LiteLLM Is (and Isn’t)

LiteLLM is an open-source proxy that sits in front of any LLM — local or cloud — and exposes a single OpenAI-compatible API endpoint. You point it at your models, it handles routing, authentication translation, rate limiting, spend tracking, and access control.

It is not a model serving framework. It does not run inference. It is the layer between your models and everything that needs to call them.

The analogy that resonates with our hardware background: LiteLLM is to LLMs what a managed network switch is to servers. The servers still run independently. The switch makes them look like a single, manageable resource to the rest of the organization.

## Two NIMs and Two vLLMs Under One Roof 
Our B300 server runs four language models simultaneously: 
| Model                   | Runtime    | Port | Notes                          | 
|-------------------------|------------|------|--------------------------------| 
| DeepSeek R1 Distill 70B | NVIDIA NIM | 8173 | Reasoning tasks, code          | 
| Mistral Small 24B       | NVIDIA NIM | 8104 | Fast chat, summarization       | 
| Qwen3.6 27B             | vLLM       | 8728 | Multilingual, Vision LLM       | 
| MiniMax M2.7            | vLLM       | 8729 | Long context, MoE architecture | 

Two of these are NVIDIA NIM containers — optimized, production-ready inference with TensorRT-LLM under the hood. Two are vLLM instances running community checkpoints. Before LiteLLM, the application had to negotiate with each of these differently.

NVIDIA recently introduced the concept of the **AI Factory** — purpose-built GPU clusters that function as internal infrastructure for AI workloads the same way compute clusters function for HPC. LiteLLM is, in practical terms, the software layer that gives a single GPU server most of the organizational benefits of an AI Factory: unified API, access control, cost attribution per virtual key, and the ability to add or swap models without touching application code.

For teams already running GPU hardware, this is not a hypothetical. We built it in a 1-week sprint.

The Three-Agent Team’s First Infrastructure Sprint

The setup task was assigned as GitHub issues in the same workflow described in Part 1. Claude Code handled the server configuration and proxy setup; Copilot handled the application-side migration to the unified endpoint; the team lead reviewed both.

The implementation itself was straightforward. What was not straightforward were two operational gotchas that would have cost significant debugging time without documentation.

## The Multiplier Effect: One Endpoint, Many Consumers

The operational win from LiteLLM is not the proxy itself — it is what becomes possible once you have a single, stable, OpenAI-compatible endpoint.

**Internal ChatGPT.** [Open WebUI](https://openwebui.com/) is an open-source chat interface that connects to any OpenAI-compatible backend. Point it at `http://your-server ` with a virtual key, and every employee has a web-based chat interface backed by your own hardware — no data leaving the building, no per-seat SaaS cost, all four models available from a dropdown.

**Internal Copilot.** VS Code’s [LiteLLM Provider for GitHub Copilot Chat Extension] extension and GitHub Copilot both support custom OpenAI-compatible endpoints. A developer can configure their IDE to use Minimax M2.7or DeepSeek for code completion and chat — the same GPU that runs the meeting AI tool can power the development environment for the team building it. The endpoint is the same; the virtual API key controls which models and how much spend.

**Any OpenAI SDK client.** Python’s `openai` library, LangChain, LlamaIndex, and most modern AI frameworks speak OpenAI API natively. Once LiteLLM is running, integrating your on-premises models into any of these tools is a one-line endpoint change:

```python 
from openai import OpenAI 
client = OpenAI(base_url="http://***.***.**.**:9400/v1", api_key="your-virtual-key") 
```

The infrastructure investment compounds. Each new use case — the meeting AI tool, the internal chat interface, the VS Code extension — runs on the same four models, the same hardware, the same management surface.

The API Key as a Product

This is the angle that changes the economics for hardware vendors.

The historical progression of giving a customer access to a GPU model for evaluation has been:

1. **SSH access** — hand over server credentials, give the customer root or a restricted shell, let them run inference directly. Significant security exposure. No usage visibility. No way to limit what they can do or access.
2. **Vast.AI model** — rent a single GPU or rack at an hourly rate. Better isolation. Still requires the customer to set up their own inference stack, which introduces setup time before they can evaluate the model they actually came to test.
3. **LiteLLM virtual key** — generate a virtual API key scoped to a specific model, with optional spend limits and request rate limits. The customer calls a standard OpenAI-compatible endpoint. No server access. No setup. No exposure beyond the API surface. From the customer’s perspective, it is indistinguishable from calling a cloud AI provider — except it is running on your hardware, and you control the model.

For AMAX, we provide GPU servers and rack solutions to companies building internal AI infrastructure, this progression is commercially significant. The standard sales motion involves a prospect asking: “Can we test DeepSeek on your B300 before we commit to a $800K server purchase?” Previously, the answer involved coordinating SSH credentials, walking them through a setup process, and hoping they didn’t accidentally touch something unrelated.

With LiteLLM, the answer is: here is an API key, here is the endpoint, here is the OpenAI-compatible spec. Call it the same way you call GPT-4. Spend limit is $50. Key expires in 48 hours. This reduces the evaluation cycle from days to hours, removes the support burden from the hardware sales team, and produces a usage log that tells the sales engineer exactly what the prospect tested and how the model performed.

The logical extension: a GPU-as-a-service offering where customers purchase inference credits scoped to specific models and hardware configurations, with no server management on their end. This is not a new business model — it is how every cloud AI provider operates. LiteLLM makes it operational on owned hardware in a day.

The Co-Existence Design

One architectural decision from the implementation deserves mention because it reveals a constraint that organizations will encounter.

We wanted LiteLLM as the default endpoint for all four models, but we also needed the ability for developers to bypass the proxy and hit individual model ports directly — for debugging, for testing model-specific parameters not exposed through the proxy, for scenarios where the proxy itself is what’s being debugged.

The solution was a template-plus-override pattern. The application ships with `llm_config.template.json` defaulting all models to the LiteLLM port. An external `llm_config.json` in the user’s app data folder can override any provider with a direct-port URL. The application loads the external config if present and falls back to the template.

This means: the default experience is LiteLLM-unified. An advanced user who needs direct access to Mistral’s port 8104 creates one entry in their local config and gets it, without changing anything for anyone else.

The pattern generalizes: LiteLLM is infrastructure, not a hard dependency. Teams that adopt it should design their applications to work with or without it.

What the Sprint Taught Us

Three months ago, accessing any of our four models required knowing which port it ran on and how its authentication worked. Today, any internal tool, any team member, and any prospective customer evaluation can be served from a single endpoint with a scoped API key.

The three-agent team structure from Part 1 — Claude Code handling server-side implementation, Copilot handling application-side migration, the team lead reviewing both — turned what could have been a multi-week infrastructure project into a focused sprint with clear issue scope, a design document reviewed before any code was written, and a test suite that verified the Bearer token injection worked for all four providers before the PR merged.

The tooling compounds. The infrastructure compounds. The team structure that makes both move faster is the same one described in Part 1.

The Larger Point for GPU Hardware Vendors

LiteLLM’s role in the AI stack is still underappreciated in hardware sales conversations. The discussion tends to focus on the GPU itself — VRAM, tensor cores, NVLink bandwidth — and the model — parameter count, benchmark scores, context length. The management layer that makes the GPU a *shared organizational resource* rather than a dedicated server for one team gets treated as an afterthought.

For any organization purchasing GPU infrastructure for internal AI deployment, the procurement conversation should include: how will access be managed across teams, how will usage be attributed for cost accounting, how will evaluation access be provided to new stakeholders without exposing the full server. LiteLLM answers all three, and it is free, open-source, and deployable in a day.

The AI Factory is not a separate purchase. For many teams, it is a configuration file and a proxy on a server they already own. AMAX HostMax brings altogether in one project.