Tool Use With Claude Through a Gateway: Streaming, Errors, and Cost Patterns

Tool use is the core of every Claude agent. It is also where things quietly break when traffic flows through a gateway. This is a practical guide to the round trip, the streaming deltas, the retry boundaries, and the actual cost of one agent loop on BUZZ.

Why tool use is the part that breaks

Plain text completion through an AI gateway is forgiving. The request goes up, tokens come back down, and even a sloppy proxy that buffers responses or rewrites JSON will usually still produce something usable. Tool use is different. It is a bidirectional protocol embedded in the same HTTP stream, with strict ordering between tool_use blocks the model produces and tool_result blocks the client must return. Any layer in the middle that touches the request body, normalizes JSON whitespace, drops unknown fields, or buffers the SSE stream until completion will eventually corrupt an agent loop in a way that is hard to reproduce.

That is why BUZZ treats tool-use traffic as a transparent forwarding case. The bytes you send are the bytes that arrive upstream. The bytes upstream returns are the bytes you receive. This article documents what that contract means in practice, where the failure modes hide, and what an agent loop costs once you account for the overhead Anthropic adds when tools are present.

What the tool-use round trip actually looks like

The Anthropic Messages API expresses tool use as content blocks rather than a side channel. A normal assistant turn returns a list of content blocks, each tagged with a type of text, tool_use, or thinking. When the model decides to call a tool, the response body contains an item like this:

{
  "id": "msg_01Abc",
  "type": "message",
  "role": "assistant",
  "model": "claude-sonnet-4-6",
  "stop_reason": "tool_use",
  "content": [
    {
      "type": "text",
      "text": "I'll look that up for you."
    },
    {
      "type": "tool_use",
      "id": "toolu_01XyZ",
      "name": "get_weather",
      "input": {"city": "Tokyo", "units": "metric"}
    }
  ],
  "usage": {"input_tokens": 412, "output_tokens": 87}
}

The client is then expected to execute the tool, capture the result, and start the next request with the assistant turn replayed verbatim and a new user turn that carries a matching tool_result block:

{
  "model": "claude-sonnet-4-6",
  "max_tokens": 1024,
  "tools": [...],
  "messages": [
    {"role": "user", "content": "What's the weather in Tokyo?"},
    {"role": "assistant", "content": [
      {"type": "text", "text": "I'll look that up for you."},
      {"type": "tool_use", "id": "toolu_01XyZ", "name": "get_weather",
       "input": {"city": "Tokyo", "units": "metric"}}
    ]},
    {"role": "user", "content": [
      {"type": "tool_result", "tool_use_id": "toolu_01XyZ",
       "content": "18C, partly cloudy"}
    ]}
  ]
}

Three properties of this protocol matter for gateways. First, the tool_use_id in the result must match the id the model emitted, byte for byte. Second, the assistant turn must be replayed exactly, with all blocks in the original order, otherwise the model loses its place and may either repeat the call or hallucinate that the tool already ran. Third, the conversation grows on every turn because the entire history is replayed, which is why prompt caching becomes load-bearing in long loops.

Through a gateway: what passes through unchanged

BUZZ forwards Claude tool-use traffic without altering any field that participates in the protocol. In particular:

• The tools array, including each tool's name, description, input_schema, and any cache_control markers attached to its definition.
• The tool_choice field, in all four shapes: auto, any, {"type":"tool","name":"..."}, and none.
• Every block inside messages[].content, including tool_use and tool_result blocks, with their ids and ordering preserved.
• The full SSE event stream when stream: true, byte for byte and event boundary for event boundary.
• Beta headers such as anthropic-beta for prompt caching and computer-use experiments.

The practical implication is that any agent harness that already runs against api.anthropic.com works against BUZZ by changing one variable. With the official Python SDK that is Anthropic(base_url="https://buzzai.cc", api_key=...). If you prefer the OpenAI-compatible surface for Claude, the base URL is https://buzzai.cc/v1 and tool use is forwarded through the OpenAI tool-call shape. For Claude Code itself, the one-line installer at https://buzzai.cc/sh/claudecode.sh points the CLI at the gateway without touching project files.

Streaming plus tool use: where it usually breaks

Streaming responses are emitted as Server-Sent Events. With tool use enabled, the same stream interleaves text deltas and tool-input deltas, and the client must reconstruct each block from its events. The minimum event shapes you have to handle are:

• message_start — opens the response, carries usage.input_tokens.
• content_block_start — announces a new block at index N. The content_block field tells you whether it is text, tool_use, or thinking.
• content_block_delta — the actual incremental data. For text it is text_delta. For tool use it is input_json_delta with a partial_json string that you concatenate.
• content_block_stop — the block at index N is finished.
• message_delta — carries stop_reason and the final usage.output_tokens.
• message_stop — the response is over.

The trap is that input_json_delta chunks are not individually valid JSON. You only get a parseable object after concatenating every delta in a block and parsing the full string at content_block_stop. Naive proxies that try to "validate" JSON on the way through corrupt this stream. Naive clients that try to json.loads each chunk crash on the second character. Here is a streaming handler that does it correctly with the Anthropic Python SDK and BUZZ as the base URL:

import json
import os
from anthropic import Anthropic

client = Anthropic(
    base_url="https://buzzai.cc",
    api_key=os.environ["BUZZ_API_KEY"],
)

tools = [{
    "name": "get_weather",
    "description": "Get current weather for a city.",
    "input_schema": {
        "type": "object",
        "properties": {
            "city": {"type": "string"},
            "units": {"type": "string", "enum": ["metric", "imperial"]},
        },
        "required": ["city"],
    },
}]

# Accumulator: index -> {"type": ..., "name": ..., "id": ..., "input_buf": ""}
blocks = {}

with client.messages.stream(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    tools=tools,
    messages=[{"role": "user", "content": "Weather in Tokyo, metric please."}],
) as stream:
    for event in stream:
        t = event.type

        if t == "content_block_start":
            cb = event.content_block
            blocks[event.index] = {
                "type": cb.type,
                "name": getattr(cb, "name", None),
                "id": getattr(cb, "id", None),
                "input_buf": "",
                "text_buf": "",
            }

        elif t == "content_block_delta":
            d = event.delta
            block = blocks[event.index]
            if d.type == "text_delta":
                block["text_buf"] += d.text
                print(d.text, end="", flush=True)
            elif d.type == "input_json_delta":
                block["input_buf"] += d.partial_json

        elif t == "content_block_stop":
            block = blocks[event.index]
            if block["type"] == "tool_use":
                # Now and only now is the JSON complete.
                block["input"] = json.loads(block["input_buf"] or "{}")
                print(f"\n[tool_use] {block['name']}({block['input']})")

    final = stream.get_final_message()
    print(f"\nstop_reason={final.stop_reason}, usage={final.usage}")

Three things to note. The input_buf is only parsed at content_block_stop, never before. The id captured at content_block_start is what you must echo back as tool_use_id when you submit the result. And the SDK's get_final_message() gives you a fully assembled message object that you can append to messages as the assistant turn for the next iteration without rebuilding it manually.

Cost patterns: what one agent loop actually costs

Tool use adds two consistent overheads on top of your prompt and the model's response. Both are billed as input tokens.

• Tool-use system prompt. When the request includes a non-empty tools array, Claude prepends an internal system prompt that teaches the model how to format tool_use blocks. On Claude 4.x this overhead is 346 input tokens per request.
• Tool definitions. Each entry in tools is serialized as JSON schema and counted as part of the input. A small definition like the get_weather example above costs roughly 60 to 100 tokens. A complex schema with nested objects, enums, and rich descriptions can run 200 to 400 tokens.

To make this concrete, here is a representative single-turn call that triggers one tool: a 600-token user prompt, three tool definitions averaging 120 tokens each, the 346-token tool-use overhead, and a model response containing a 40-token text preamble plus a tool_use block whose JSON input is 30 tokens. The accounting on /api/pricing for Sonnet looks like this:

Now the second turn. The client executes the tool, gets back, say, a 250-token result, and re-sends the entire history plus the tool_result block. The same 346-token overhead and 360-token definitions are charged again because they are part of every request that has tools set. The new input is the original 600-token user prompt, the 70-token assistant turn replayed verbatim, and the 250-token tool_result, for an input subtotal of 1626 tokens. If the model now answers in 180 output tokens, the full two-turn loop has cost 1306 + 70 + 1626 + 180 = 3182 tokens at Sonnet rates.

Two practical levers reduce that. First, add a cache_control breakpoint at the end of tools; on the second turn the 346 + 360 = 706 tokens of overhead read from cache at 0.1x the base input rate instead of full price. Second, mark the system prompt as cacheable too if you have one. For long agent loops with five to ten iterations, those two breakpoints typically cut the input bill in half. The exact multipliers and the live per-token rates are published on https://buzzai.cc/api/pricing; the gateway forwards cache_control markers unchanged, so caching works the same as it does directly against Anthropic.

Error handling and retry boundaries

An agent loop has two retry boundaries that do not move together. The first is the API call to the gateway. The second is the local execution of the tool itself. Conflating them is the most common source of duplicate side effects, such as sending a notification twice or charging a card twice when something half-failed.

Treat each boundary independently:

• Upstream 5xx (502, 503, 504, 529). These come from the gateway or upstream. The request was either never executed or executed with no client-visible side effect. Retry with exponential backoff. Three attempts with 1s, 3s, 9s jitter is a sensible default.
• 429 rate limit. Honor retry-after if present, otherwise back off the same way as 5xx. BUZZ surfaces upstream rate-limit headers untouched.
• 4xx other than 429. Do not retry. A 400 means the request was malformed, often because of a missing tool_use_id or a tool_result that does not match an emitted tool_use. Fix the message construction, not the network.
• Tool execution failure. Do not retry the call to Claude. Instead, return a tool_result with is_error: true and a short message. The model will adapt on the next turn, often by trying different arguments or asking the user.
• Tool execution timeout. Same shape: kill the local task, return {"type":"tool_result","tool_use_id": "...", "is_error": true, "content": "Tool timed out after 30s"}. The conversation continues; the model decides what to do.
• Malformed JSON in tool input. Rare on Claude 4.x but possible when the model is asked to embed large strings. Catch json.JSONDecodeError when parsing the assembled input_buf, and return an is_error tool_result describing the parse failure. The model self-corrects on the next turn.

The discipline that ties this together: a tool execution failure is data the model needs, not an exception to escalate. Surfacing it as a structured tool_result keeps the conversation in a valid state. Throwing away the assistant turn and silently retrying is what produces duplicate charges and confused agents.

A realistic agent loop

Here is a complete agent loop, around 50 lines, that you can paste into a script. It connects to BUZZ, registers two tools, runs the conversation until the model stops calling tools, and enforces both a turn cap and a token cap. The body is deliberately compact so you can read it end to end.

import json
import os
from anthropic import Anthropic

client = Anthropic(base_url="https://buzzai.cc", api_key=os.environ["BUZZ_API_KEY"])

TOOLS = [
    {"name": "get_weather", "description": "Current weather for a city.",
     "input_schema": {"type": "object", "required": ["city"],
        "properties": {"city": {"type": "string"}}}},
    {"name": "search_docs", "description": "Search internal documentation.",
     "input_schema": {"type": "object", "required": ["q"],
        "properties": {"q": {"type": "string"}, "limit": {"type": "integer"}}}},
]

def execute_tool(name, args):
    try:
        if name == "get_weather":
            return f"{args['city']}: 18C, partly cloudy"
        if name == "search_docs":
            return f"Top result for {args['q']!r}: see internal-runbook#42"
        return {"is_error": True, "content": f"unknown tool {name}"}
    except Exception as e:
        return {"is_error": True, "content": f"{type(e).__name__}: {e}"}

def run_agent(user_prompt, max_turns=15, token_budget=200_000):
    messages = [{"role": "user", "content": user_prompt}]
    total_tokens = 0
    for turn in range(max_turns):
        resp = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=2048,
            tools=TOOLS,
            messages=messages,
        )
        total_tokens += resp.usage.input_tokens + resp.usage.output_tokens
        if total_tokens > token_budget:
            raise RuntimeError(f"token budget exceeded: {total_tokens}")

        # Replay the assistant turn verbatim.
        messages.append({"role": "assistant", "content": resp.content})

        if resp.stop_reason != "tool_use":
            return resp, messages

        # Build a single user turn with one tool_result per tool_use block.
        results = []
        for block in resp.content:
            if block.type != "tool_use":
                continue
            output = execute_tool(block.name, block.input)
            if isinstance(output, dict) and output.get("is_error"):
                results.append({"type": "tool_result", "tool_use_id": block.id,
                                "is_error": True, "content": output["content"]})
            else:
                results.append({"type": "tool_result", "tool_use_id": block.id,
                                "content": str(output)})
        messages.append({"role": "user", "content": results})
    raise RuntimeError(f"agent did not converge within {max_turns} turns")

if __name__ == "__main__":
    final, history = run_agent("What's the weather in Tokyo, and find docs on rate limiting.")
    text = next((b.text for b in final.content if b.type == "text"), "")
    print(text)

A few details worth pointing out. The assistant turn is appended as resp.content directly, the SDK objects round-trip cleanly back into the next request. Each tool_use block in a single assistant turn produces exactly one tool_result block in the following user turn, in the same order, all bundled into one user message. execute_tool never raises out of the loop; failures become structured is_error results so the conversation stays valid. The two caps, max_turns and token_budget, exist to bound runaway loops where the model keeps calling tools or where one tool returns a 50k-token blob that the model keeps re-reading.

To run this against the gateway with no other change, the base_url is already set. The BUZZ_API_KEY can be obtained from your account on https://buzzai.cc/, and the available model identifiers are listed on https://buzzai.cc/models.

FAQ

Conclusion

Tool use is where agent systems earn or lose their reliability. The protocol is small but unforgiving: tool_use_id must round-trip exactly, the assistant turn must be replayed verbatim, the SSE stream must not be buffered, and the JSON in input_json_delta must be assembled before parsing. A transparent gateway does not get in the way of any of that. BUZZ forwards the bytes, preserves the streaming contract, retains nothing about the content, and prices tool-use overhead the same way Anthropic does, on top of a discounted base rate.

If you are starting from scratch, the agent loop above is a complete reference. If you already have an agent harness, switching to the gateway is one variable change, base_url="https://buzzai.cc" in Python, or the equivalent in TypeScript, and Claude Code via https://buzzai.cc/sh/claudecode.sh. Live per-token rates are on /api/pricing and the active model list is on /models.