OpenAI-Compatible APIs Still Break at the Tool-Call Step

The test case looked modest: an order-routing assistant needed to call `get_inventory` and `quote_shipping` before recommending where to fulfill an order. Both calls could run at once because neither depended on the other, and the assistant then received both results and wrote one customer-facing answer.

Ordinary chat worked across the supposedly compatible endpoints in the evaluation. The same messages array went in, authentication followed the expected pattern, and text came back in a familiar response envelope. The trouble started at the next step, when the model had to return structured instructions rather than prose.

That distinction matters because “OpenAI-compatible” usually describes a request surface, not complete behavioral conformance. An endpoint may accept `POST /v1/chat/completions`, recognize roles such as `system` and `user`, and return text under a familiar field while interpreting tool definitions, streaming events or follow-up messages differently.

The order assistant exposed four breakpoints. Any one of them can turn a successful model migration into failed orders, repeated tool execution or an agent that quietly falls back to guessing.

Parallel calls are the first meaningful test

A tool call is a model-generated request for application code to run a named function with structured arguments. In this case, the first user message supplied an order identifier and destination, while the request advertised the inventory and shipping tools with their argument schemas.

The useful response was two independent calls in one assistant turn. The application could execute them concurrently, return both outputs, and ask the model to finish. That makes parallel calling newly practical for workflows in which latency is dominated by external systems: inventory databases, carrier services, search indexes or internal APIs.

Compatibility at the chat layer did not guarantee that behavior. The harness had to distinguish an endpoint that emitted two calls from one that selected only one tool, waited for its result and requested the second later. Both paths could eventually produce an answer, but the serial path added another model round trip and another opportunity for the conversation state to drift.

A more damaging variation was structural. A response could resemble the expected tool-call object while omitting a stable call identifier, changing the field that held the function name or returning arguments in a form the client library did not parse. The model had made a reasonable decision, yet the application treated the turn as malformed.

For the order assistant, the adapter’s first job was therefore normalization. It converted each provider response into an internal record containing a call ID, tool name and argument object. If the provider supplied no usable ID, the adapter generated one and retained the provider’s original value for the follow-up turn. It also preserved call order without assuming that order carried meaning.

Do not test this with two dependent tools. If the second tool needs the first tool’s output, a model that emits only one call has behaved correctly. Use independent operations, as the inventory and shipping checks were, then assert that the response contains two executable calls or document that the endpoint will run them serially.

Accepted schemas can still mean different arguments

The next fault line was the argument schema, the JSON Schema description that tells a model which fields and values a tool accepts. The shipping function required a destination and service level; the inventory function required a product identifier, with no undeclared properties allowed.

An endpoint accepting the `tools` field had not necessarily honored every schema constraint. The practical test was not whether the HTTP request returned an error. It was whether the generated arguments survived validation before any business function ran.

Nested objects, enumerated values and `additionalProperties: false` deserve separate probes because providers may support different subsets of JSON Schema. A client that assumes full enforcement can receive an invented field, a value outside the allowed set or a string where the function expects an object. Permissive application code may then conceal the incompatibility until a rarer input reaches production.

The adapter should parse arguments once, validate them against the application’s own schema and reject anything outside it. Provider-side schema enforcement is useful, but it is not the security boundary. For the order assistant, an invalid service level triggered a bounded repair turn that included the validation error and the allowed values; it did not pass the model’s original string to the carrier integration.

That repair has a price. It consumes another inference request, delays the answer and can fail again, so teams should record repairs as compatibility failures rather than celebrating the eventual response as a pass. A migration that doubles the number of model turns on schema-heavy requests may erase the price advantage that motivated it.

There is also a design tradeoff. Flattening a complicated schema into a few strings often improves portability, but it moves parsing and validation into application code and gives the model less guidance. Keeping the richer schema reduces glue code on endpoints that support it. The order assistant used the smallest schema that represented the business rule without asking the model to encode an entire order object.

Streaming breaks in the middle, not at the request

Streaming, which delivers a response in incremental events rather than one finished object, created the least visible failures. Text streaming encourages clients to append each fragment as it arrives. Tool-call streaming needs more state because the function name, arguments and call identifier may arrive in separate deltas, and fragments for parallel calls may be interleaved.

The order assistant’s two calls made that difference easy to see. A parser that concatenated every argument fragment into one buffer produced invalid JSON when inventory and shipping deltas alternated. A parser keyed only by arrival position could attach later fragments to the wrong call if an endpoint identified them differently from the reference behavior.

The safe approach was to accumulate fragments by the provider’s call index or identifier, delay JSON parsing until the call finished, and validate the completed object before execution. The adapter also treated a stream ending without a recognized completion signal as incomplete, even when the accumulated text happened to parse. Executing early reduces latency by a small amount but risks running a tool from a truncated or subsequently revised argument payload.

This is where an integration can pass every non-streaming test and still fail under the production setting chosen to improve responsiveness. Run the same prompt with streaming disabled and enabled, then compare normalized calls rather than raw event shapes. If the results differ, streaming needs its own provider implementation instead of a shared parser with extra conditionals scattered through the agent loop.

Logging must happen before normalization as well as after it. The raw event sequence explains whether the provider, software development kit or local adapter lost a fragment; the normalized record shows what the application believed it received. Without both, a malformed shipping call looks like model unreliability even when the defect sits in stream assembly.

Tool results complete the protocol

The final test returned inventory and shipping data to the model. This is where basic compatibility claims become especially weak, because the endpoint must understand a multi-turn transcript containing the assistant’s tool requests and one result message for each call.

The call ID links a result to the request that produced it. If an endpoint omits IDs, rewrites them or expects a different role and field layout for tool output, replaying the reference transcript may fail validation or leave the model unable to associate the stock count with the correct function. Converting the result into an ordinary user message can keep the conversation moving, but it discards protocol structure and makes prompt injection inside tool output harder to isolate.

For the order assistant, the adapter stored an internal transcript rather than treating any provider’s wire format as the system of record. On each turn, it rendered that transcript into the endpoint’s required message shape. Tool outputs remained labeled as untrusted data, call IDs stayed attached, and a provider switch did not require rewriting historical messages already held by the application.

That setup is more work than changing a base URL. It is also the point at which switching becomes operationally practical: the agent loop can keep one execution model while small serializers and stream parsers absorb endpoint differences.

A sensible acceptance gate has four cases tied to the same workflow. Require two independent calls, validate intentionally constrained arguments, repeat the request over streaming, then return both results and demand a final answer with no extra tool execution. Capture raw requests and responses, but redact credentials and sensitive tool data before retaining them.

Basic chat success remains useful. It shows that authentication, model selection and ordinary message handling are close enough for low-risk text generation. It does not clear an endpoint for an agent that can touch inventory, send email or modify records. The order-routing run only passed when the whole loop completed, not when the first assistant response looked familiar.

Questions people ask

What does OpenAI-compatible usually guarantee?

It commonly means an endpoint accepts a familiar URL pattern, authentication method and chat-completions request shape. It does not by itself guarantee identical tool schemas, streamed event layouts, parallel-call behavior or tool-result messages. Check the provider’s documentation and run a full transcript through the exact client version used in production.

Can

I use the same OpenAI client library with another endpoint?

Often, for basic text chat. Tool use may still require custom serialization, stream assembly or response normalization behind that client. Pointing the library at a new base URL is an initial connectivity test; it is not evidence that a multi-step agent will preserve call IDs, arguments and result associations.

Should an application execute tool calls before a stream finishes?

Usually no. Buffer each call separately, wait for a recognized completion condition, parse the finished arguments and validate them before execution. Early execution can save part of a response interval, but a truncated stream or interleaved parallel call can send incomplete data to a real system and create duplicate-work risks during retries.

What is the minimum useful compatibility test?

Use one repeatable workflow with two independent tools and constrained schemas. Run it with and without streaming, return both tool results, and require one final answer without repeated calls. For the order assistant, that sequence tested the protocol boundary that a plain chat prompt never reached.