AI Gateway

What is the challenge?

How do you control cost, latency, and observability across multiple AI providers?

Production AI apps quickly hit problems: per-call costs are unpredictable, you have no visibility into which prompts are expensive, repeated identical prompts cost money every time, one provider has an outage and your app goes down, and rate limits hit at the worst moments. The native solutions are each vendor's bespoke dashboard — which doesn't help when you're using three vendors. You need one control plane in front of all of them.

Architecture

How it fits together

Diagram coming soon

Architecture diagram for this solution will be added here.

Cloudflare's Solution

How Cloudflare solves it

Unified endpoint for every provider. Point your code at a single Cloudflare AI Gateway URL and it routes to OpenAI, Anthropic, Workers AI, Google AI, Azure OpenAI, Mistral, Replicate, AWS Bedrock, or any compatible endpoint.
Response caching. Identical prompts return cached responses — configurable per-route. Production apps with repeated queries cut spend 30–70% with caching on. Optional semantic caching (cache near-identical prompts) is available too.
Rate limiting & spend controls. Set per-app, per-key, per-user rate limits and budget caps. Stop runaway prompts and accidental fan-outs before they melt your bill.
Fallback & retries. Route primary requests to OpenAI; on error or timeout, transparently fall back to Anthropic or Workers AI. Configurable per-route, per-model. Prevents single-provider outages from breaking your app.
Full request/response logging. Every prompt and completion logged with token counts, latency, cost, and metadata. Searchable, exportable, attributable. The 'what is my AI actually doing' problem solved.

FAQ

Common questions

Does AI Gateway add latency?

Minimal — typically a few milliseconds for the routing decision. For cached requests, it actually *reduces* latency (the cached response is served from Cloudflare's edge instead of waiting for an OpenAI round-trip). For non-cached requests, the Gateway is on the same Cloudflare network that's already in the path, so the overhead is negligible.

Can I bring my own API keys (BYOK) or do I need to use Cloudflare's?

BYOK. AI Gateway forwards your existing OpenAI/Anthropic/etc. keys — Cloudflare doesn't bill you for those provider charges; you keep your provider relationship. Cloudflare bills only for AI Gateway features (caching, logging, etc.), and the Workers AI models you choose to run.

Does the cache understand 'similar' prompts, or only exact matches?

Both. Exact-match caching is on by default (same prompt + same parameters = cached). Semantic caching (similar prompts return the same cached response based on embedding similarity) is configurable per-route — useful for FAQ-style applications.

What if I want to switch from OpenAI to Workers AI later?

Change a config in AI Gateway. Your app code doesn't change. Universal endpoint format means swapping models or providers is one route-config change.

Let's Test!

Try it live

Every message in the chatbot widget on this page routes through Cloudflare AI Gateway to Workers AI (Llama 3.3 70B). Send a test prompt below or open the chatbot — both share the same Gateway.

This entire site uses AI Gateway

Open the chat widget in the bottom-right corner — every message is routed through Cloudflare AI Gateway to Workers AI. The same gateway handles caching, rate limiting, request logging, and (in production configs) provider fallback.

Route: browser → /api/chat → AI Gateway → workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast

Send a test prompt through the gateway

Response

Error

Tip: send the same prompt twice. If AI Gateway response caching is on for this route, the second call returns much faster.