Defining the LLM Adapter — Architecture and Technical Considerations

Open any codebase that calls an LLM and you will find the seam where the provider gets swapped. The names vary: LLMClient, ModelBackend, Completion, or sometimes just a bare chat(). What is surprisingly rare is an explicit statement of what that layer owns and what it does not. With one provider there was no reason to settle the boundary, and by the time the second provider arrives it is already too late, because if provider == "..." has seeped into the call path in a dozen places.

This post defines that layer, the LLM adapter, as precisely as I can, and then works through the decisions that actually come up when OpenAI, Anthropic, Google and a self-hosted model served by vLLM all have to sit behind it. The examples are mostly things we learned building this layer into an agent runtime that runs inside a MicroVM, and then verifying it against real endpoints.

Definition: an adapter normalizes semantics, not data

An LLM adapter is the component that maps bidirectionally between an application's provider-neutral request and response model and a specific provider's wire protocol, normalizing not only the happy-path data structures but also the failure semantics, the streaming semantics and the round-trip invariants.

The second half of that sentence is the part that matters. Moving JSON field names around is the easiest and least important thing an adapter does. The code that builds a request body gets written once and then barely changes. What actually stands in the way of an adapter behaving predictably is the rate limit that arrives under a status code other than 429, the error delivered after a 200, the rejection that only shows up on the second turn. The value of the adapter lies in absorbing those so that nothing above it has to know they exist.

Laying out what the adapter is responsible for, then, it divides into five parts.

• Request mapping — serializing a provider-neutral request into the request body a particular provider expects. Where the system instruction goes, how tools are declared and what the parameters are called all belong here.
• Response normalization — converting the provider's reply into the neutral response shape and handing it back. Text, tool calls, the stop reason and token usage are what that response carries.
• Error classification — sorting a failure into categories the caller can decide how to act on.
• Stream reassembly — with streaming on, a reply does not arrive complete in one piece; it flows in as finely chopped events. The adapter turns those pieces into neutral deltas and then reassembles them into exactly the response the buffered path would have produced. The OpenAI format, for instance, splits a tool call's arguments into string fragments such as {"pa · th": "/e · tc"} and tags each fragment with nothing but an index, so the call only becomes whole once the adapter appends each fragment to a per-index buffer. Get that reassembly wrong and merely changing the streaming setting changes the conversation history.
• Opaque state passthrough — some of what a provider attaches to a response is something the adapter has no need to understand yet must send back untouched on the next request. The adapter stores such a value in the neutral model without interpreting it and returns it verbatim on the following turn. The clearest case is the thought_signature Gemini 3 attaches to every function call: send the tool results back without it and the request itself is rejected with a 400.

The last one needs a more concrete explanation, so it gets a section of its own below. It is also the item that tends to be discovered last and to carry the highest cost of correction.

Conversely, it is worth listing the things that are easily mistaken for the adapter's job but that it deliberately does not do.

• Retry and backoff policy — the adapter classifies a failure and passes it up; it does not decide whether to try again or how long to wait. How many seconds to hold off on a rate limit and how many attempts to make is a question for the application, not a property of the provider. Put that policy inside the adapter and switching providers silently changes the retry count and the delays along with it, a difference that appears in no configuration file, so the same job fails differently depending on which provider ran it.
• Tool execution — the adapter hands up a normalized tool call. It does not run it.
• Conversation state and context management — keeping, compacting and summarizing history belongs above.
• Model selection and routing — which model to send to is a matter of policy, not something the protocol deals with.

Once the adapter's role is pinned down this way, it comes to resemble a plain function call: the input is a neutral request, the output is a neutral response or a classified error, and nothing in between spends time on a judgement.

The adapter's boundary comes into sharper focus against three neighbouring things. An SDK wrapper adds call-site convenience to the client a vendor ships; the provider's own request and response types stay visible to the caller, so changing providers changes the caller's code as well. A gateway or proxy performs the same five surfaces, but performs them in a separate process across the network, which puts its availability and its version beyond the control of your deploy and leaves credentials and call records outside your process. A framework's model abstraction contains an adapter but also prescribes prompt construction, call chaining and conversation memory, mandating a great deal more than protocol translation. Set against those three, an adapter is the layer that exposes no provider type to its caller, runs inside the application's own process, and prescribes nothing about usage beyond protocol translation.

Reviewing the two adapter architectures

Once you commit to supporting several providers there are really only two options.

Option A is to own a native client per provider. You speak whatever each vendor documents as its official API, so fidelity is highest and vendor-specific features are immediately available. The cost scales with the number of providers, and counting it in lines of code badly understates it. The real cost is the number of combinations you have to verify: N providers times M scenarios, doubled again because streaming and buffered paths have to be checked separately. On top of that, an adapter is not the kind of code whose correctness you can settle by reading it. No amount of studying the specification tells you whether the server will accept the request you just built; that is only established by sending it to the real endpoint and looking at the reply.

Option B is to pick one wire format and reduce a provider to a base URL. This became practical once OpenAI's chat-completions format turned into the de facto interoperability format. Groq exposes api.groq.com/openai/v1, Google exposes generativelanguage.googleapis.com/v1beta/openai, and vLLM, llama.cpp's server and Ollama all serve the same shape out of the box. This changes the very nature of what a new provider costs. Under option A each new provider adds a whole further round of implementation and verification; under option B it adds one more address to a list. There is a single client throughout, and the provider presets are nothing more than a list of named base URLs.

The price of option B is clear enough. You inherit the fidelity of a compatibility layer you do not own, and vendors differ widely in how far they stand behind theirs. So the question to ask is not whether the format matches. It is whether the vendor stands behind that compatibility path as a supported production route, or has opened it only so you can try out and compare models.

Apply that test and the three commercial providers land in different places. What follows, though, reflects the vendor documentation as published in August 2026, when this was written; the standing of a compatibility layer can change with vendor policy, so check the current documentation before committing to one.

Google publishes its OpenAI-compatible endpoint as a product surface, so it joins by adding one base URL, and Groq is the same. Anthropic does offer an OpenAI SDK compatibility layer, but presents it as a way to try out and compare models rather than as the supported production path. The distinction looks minor and changes the decision completely: supporting Anthropic means owning another native adapter, not adding another base URL. That adapter has to implement all five surfaces from scratch, and as the next section shows, Anthropic's native format differs from OpenAI's in the structure of the conversation itself, not merely in field names.

One practical corollary. Even if you choose option B, design the neutral model so that option A can be absorbed later. A neutral model that simply borrows the OpenAI request shape is not a neutral model; it is the OpenAI model under a different name, and the day a native adapter shows up the whole neutral layer has to be rewritten.

Designing the neutral model

Look at the differences the neutral model has to absorb and the design guidance falls out on its own.

The system instruction lives in a different place per provider. In the OpenAI format it is the first element of the messages array; in Anthropic's API it is a separate top-level parameter; in the native Gemini API it is system_instruction. So a neutral request has to carry the system instruction separately from the message list. Suppose the neutral model has no dedicated field and stores the system instruction as one element of the message array instead. Building an Anthropic request, the adapter now has to search that array for the element that belongs in the top-level parameter, and the array may also hold other system-role messages inserted mid-conversation, with nothing left to tell the two apart. The adapter ends up inventing a rule such as "treat the first system message as the system instruction", and the moment that assumption does not hold, the wrong text takes the system instruction's place.

Tool results have a different role. In the OpenAI format a tool call is an entry in the assistant message's tool_calls array and its result comes back as a message under a role of its own, role: "tool". In Anthropic's format a tool call is a tool_use block inside the assistant's content array and its result is a tool_result block carried by the next user turn. On one side the tool result is a first-class role; on the other it is part of a user turn. If the neutral model makes the tool role first-class, the Anthropic adapter folds it into a user turn; if it does the reverse, the OpenAI adapter unfolds it. Either way somebody folds, so the thing that matters is being explicit that the folding happens inside the adapter.

Tool arguments are encoded differently. The OpenAI format ships arguments as a JSON string; Anthropic ships an already-parsed object. Unpacking a string into an object and packing it back into a string looks harmless, but the content survives the trip while the bytes do not: keys come back in a different order, whitespace disappears, and number formatting and the escaping of non-ASCII characters no longer match the original. The catch is that prompt caching only hits when the leading bytes of the request match exactly. An implementation that re-serializes arguments every time it replays the history produces slightly different bytes on each pass, missing the cache with no way to notice it is doing so. Keeping arguments in the neutral model as the bytes they arrived in is the safer default.

The vocabulary of stop reasons differs. stop, length and tool_calls versus end_turn, max_tokens and tool_use are different names for the same events. Normalize them, but do not throw the original string away. When you are reconstructing an incident from logs, what you need is the name the server used, not the name you assigned.

Reflect those four and the neutral model converges on roughly the same shape everywhere: the system instruction sits apart from the messages, a message carries a role plus text plus tool calls plus tool results, and a tool call carries an id, a name and the original argument bytes. Most implementations get this far. The field that actually causes production incidents, however, is not on that list.

Normalization is lossy: make room for opaque state

Normalization is by definition the folding of many representations into one, and whatever gets folded away is gone. The problem arrives when a provider requires you to hand back, on the next turn, something it attached on this one.

The most representative example is Gemini 3's thought_signature. The model returns a signature with each function call, and if that signature is not present in the follow-up request carrying the tool results, the request is rejected with a 400. The shape of this failure is what makes it interesting.

• The first turn looks perfectly healthy. Ask for text and text comes back; provoke a tool call and a tool call comes back.
• What breaks is the second turn, the one that returns the tool results — which is to say, everything an agent actually does.
• A smoke test that makes one call and checks the reply will therefore never catch it.

The signature attached to Anthropic's extended thinking blocks belongs to the same family, and there will be more of these over time: the harder a model works to verify the integrity of its own intermediate state, the more reasons it has to ask the client to hold and return something.

As long as providers differ this widely in what they attach and what they demand back, handling each kind of value on its own terms means revisiting the neutral model every time a new provider appears. The way to push that complexity outside the adapter is to stop trying to know what the value is at all. Concretely: at every place a provider can attach something — the tool call, the message and the content block alike — keep one field that the adapter stores and never reads. When it parses a response the adapter lifts whatever sat in that slot into the neutral model untouched, and puts it back in the same slot, exactly as received, when it builds the next request.

type ToolCall struct {
    ID   string
    Name string
    Args json.RawMessage
    // Data the provider attached and requires back, verbatim, on the next turn.
    // The adapter never interprets this; it only round-trips it.
    ProviderMeta json.RawMessage
}

One condition comes attached to the rule: the moment you look inside the field and branch on what you find, every change to the provider's format breaks your code along with it, which is why the value of this field lies not in what it holds but in never being read.

Failure semantics: a status code is not a classification

Error handling is where an adapter earns most of its keep, and the first lesson of the area is that HTTP status codes cannot be used as the classification.

The first row is what Groq actually does: exhausting the free tier's tokens-per-minute budget comes back as a 413. Classify that on the status code alone and the layer above starts compacting the conversation instead of waiting a moment. It throws away a perfectly good conversation, and the next call still fails for the same reason. The third row is what Google actually does: a bad API key arrives as 400 INVALID_ARGUMENT with the message "Please pass a valid API key", so an implementation that treats every 400 as a malformed request reports a credential problem to the operator as a request-format problem.

That yields the test that separates a good classification scheme from a bad one: a classification that does not change the caller's behavior only makes the logs look tidier, so the error kinds are worth splitting exactly as far as the right-hand column above splits.

Two things are worth checking about retry hints as well.

First, do not assume the hint is in a header. The standard route is Retry-After, but Google returns a RetryInfo entry inside error.details[] whose retryDelay states exactly how long to wait, and frequently omits the header. An implementation that only reads headers discards the number the server gave it and falls back to a generic exponential backoff.

Second, a server-specified delay deserves to be respected. Clamp it down to the client's backoff ceiling and you go straight back into the same quota window and get rejected again. Keep the ceiling as a guard against a malicious or malformed value, and honor anything in a sane range.

The outer structure wrapped around the error body deserves a check too. Google's OpenAI-compatible endpoint sometimes adds one more layer, returning errors inside a top-level array as [{"error": {...}}]. Read on the assumption that the response is an object, that does not fail to deserialize; it simply fills in no field at all. The error message then becomes a dump of the whole response body, and the wait the server specified disappears. It is a silent degradation that raises no exception and leaves no trace in the logs, and defects of this kind are not found without calling the real endpoint.

Streaming: it has to reassemble into the same thing

Adding token streaming roughly doubles the adapter's surface area, and the invariant to protect reduces to one sentence. A streamed run and a buffered run must converge on the same conversation history. Break that and a setting that turns streaming on and off is changing the meaning of the conversation, which is exactly where irreproducible bugs come from. In practice it means converting the provider's incremental events into neutral deltas and keeping an accumulator that reassembles a sequence of deltas into the same structure the buffered path returns.

The specific things to watch for:

• Tool arguments arrive in fragments. The OpenAI format streams function arguments as string pieces, and the only thing saying which piece belongs to which call is an index value; the name and id are usually present only on the first piece. Other implementations deliver a complete call in one event. The accumulator has to reassemble both into the same result.
• Never put a whole-request timeout on a stream. It cuts off a healthy long generation, one with tokens arriving steadily for minutes, right in the middle. What you want is a deadline on the response-header phase plus a watchdog on how long the body goes without producing a byte. The thing to measure is the silence, not the total duration.
• Errors arrive after the 200. When quota runs out mid-stream the error comes down as an ordinary data frame. Deserialized as if it were an ordinary chunk it has no choices, no usage and no finish_reason, so it is skipped and the stream simply ends: a rate limit looks like an empty answer. In-band errors have to go through the same classifier as the buffered path.
• Whether to retry turns on whether output has already gone out. A failure before the first token can be retried exactly as in the buffered path. Once tokens have reached the user's screen, a retry duplicates output, and it is better to salvage the accumulated partial text as the answer and report the interruption separately. A half-streamed tool call should be dropped, since its arguments may be incomplete JSON.

Self-hosted models: swapping the base URL is only half the story

Putting a vLLM-served model behind the same adapter as a commercial provider is the biggest practical win of option B. The wire format is identical, so nothing about the client changes, and a self-hosted model, a locally run model and a deterministic test double all drop straight into the commercial service's slot. But the same format does not imply the same capabilities.

• There may be no auth at all, and then you must send no header. An Authorization: Bearer with an empty credential is rejected as malformed by some servers, so an empty key needs a branch that omits the header entirely rather than sending a blank one.
• The model identifier depends on how the server was launched. There is no stable published catalogue as there is with a commercial provider; it is whatever the operator started. Querying /v1/models beats sending a wrong name and failing.
• Tool calling is a conditional feature. Automatic tool choice in vLLM requires launching the server with --enable-auto-tool-choice and a --tool-call-parser appropriate to the model. Assume it works because it is in the spec and you end up watching the model print tool names as prose while your tool declarations go unused.
• There are extensions outside the spec, and fields outside it too. Guided decoding for structured output is a server-specific extension, and the reasoning_content that reasoning models emit is not an OpenAI field at all. The latter is especially awkward: replay it verbatim in the history and some servers reject the request, so the round trip itself breaks.
• Parameter names are in motion. OpenAI replaced max_tokens with max_completion_tokens in chat completions, and the reasoning models require the latter, while a great many compatible servers still only understand the former. One client speaking to both means absorbing that difference somewhere.

The principle here is to check for capabilities rather than assume them, and to fall back to a lesser behavior openly when one is missing. The worst possible handling is silently ignoring an unsupported parameter: the request returns 200, only the result differs from what you expected, and finding out why takes days.

Verification: an adapter is not proven by unit tests

Every defect described in this post has something in common. All of them were found by calling a real endpoint, and none of them were caught by unit tests against a fake server. The reason is straightforward. Fake-based tests prove that the client is consistent with itself. They cannot prove that a service accepts what you send. The fake is built from your understanding, so wherever your understanding was wrong, the fake is wrong in exactly the same way.

An adapter therefore needs a conformance harness that runs against real endpoints. Three design requirements matter.

Keep the scenario matrix minimal, but always include the round-trip turn. A simple completion, a tool call, a second turn that returns the tool results, a streamed call, a bad key and an oversized context are enough. The third one is the load-bearing case: signature round-trip defects are invisible on the first turn, so a matrix without a round-trip turn passes them straight through.

Design cost and trigger conditions into it. Keep the prompts deliberately tiny and make the run an explicit opt-in. A harness that fires merely because credentials happen to be configured makes every routine test run cost money and burn through free-tier limits. This is not a continuously running regression suite; it is the evidence you collect when an endpoint or a model changes.

Report as a matrix. Being able to see, on one screen, which scenarios passed for which provider is what makes the result actionable. A report that maps which combinations work beats one that aborts on the first failure, because support is not a boolean, it is a matrix.

In summary

Understood as a thin wrapper around a provider API, this layer looks like tedious wiring work. What it actually does is reduce the different ways several services express failure, delay and round-trip obligations into one set of rules the layers above can handle. How completely the layers above can stay ignorant of provider names is the measure of how finished the adapter is.

Compressed into a checklist:

• Keep the five surfaces (request mapping, response normalization, error classification, stream reassembly, opaque state passthrough) inside the adapter, and keep retry policy, tool execution and conversation state outside it.
• Have the neutral model carry the system instruction apart from the messages, keep tool arguments as the bytes they arrived in, and put an uninterpreted field at every place a provider can attach something.
• Classify errors by the caller's action rather than by status code, and do not invent a classification that changes no behavior.
• Look for retry hints in both the headers and the body, and never clamp a server-stated delay down to a client-side ceiling.
• Make the streaming and buffered paths converge on the same history, time streams by silence rather than by total duration, and handle in-band errors that arrive after the 200.
• Probe compatible endpoints for capabilities instead of assuming them, and degrade explicitly rather than silently ignoring what is unsupported.
• Run an opt-in conformance matrix against real endpoints, and make sure that matrix contains a tool-result round-trip turn.

Most of this list looks unnecessary while there is only one provider. It reveals itself one item at a time with the second and third, and all at once when a self-hosted model joins. Restructuring the neutral model at that point carries a far higher cost than adding one adapter, so if nothing else, settle the boundary and add the uninterpreted fields from the start.