Relying on a single provider means that an outage, a price increase, or a model change becomes a problem in your product. The fix: a thin layer between you and the providers that exposes one interface to the rest of your application. This is how we built the Mit assistant on this very website.

The principle: a small interface, many implementations

Your application needs just one function: take the conversation history and the system instructions, and return text. Each provider is implemented behind that function and selected with an environment variable.

python
import os
from typing import Callable

Message = dict[str, str]  # {"role": "user" | "assistant", "content": "..."}
Provider = Callable[[str, list[Message]], str]  # (system_prompt, messages) -> reply

def call_anthropic(system: str, messages: list[Message]) -> str:
    import anthropic
    client = anthropic.Anthropic(api_key=os.environ["AI_API_KEY"])
    response = client.messages.create(
        model=os.environ["AI_MODEL_NAME"],
        max_tokens=600,
        system=system,
        messages=messages,
    )
    return response.content[0].text

def call_openai_compatible(system: str, messages: list[Message]) -> str:
    from openai import OpenAI
    client = OpenAI(
        api_key=os.environ.get("AI_API_KEY") or "not-needed",
        base_url=os.environ.get("AI_API_BASE") or None,  # Ollama / vLLM / OpenAI
    )
    completion = client.chat.completions.create(
        model=os.environ["AI_MODEL_NAME"],
        messages=[{"role": "system", "content": system}, *messages],
    )
    return completion.choices[0].message.content

PROVIDERS: dict[str, Provider] = {
    "anthropic": call_anthropic,
    "openai": call_openai_compatible,
    "ollama": call_openai_compatible,   # Ollama exposes an OpenAI-compatible API
    "custom": call_openai_compatible,
}

Automatic fallback

Any external provider can fail: a timeout, a 429, a 5xx. Make the last link in the chain local and network-free (a knowledge base or canned reply templates) so users never see an error page.

python
import logging
logger = logging.getLogger(__name__)

def generate_reply(system, messages, user_text):
    name = os.getenv("AI_PROVIDER", "knowledge_base")
    provider = PROVIDERS.get(name)
    if provider:
        try:
            return provider(system, messages), name
        except Exception:
            logger.exception("provider %s failed, using fallback", name)
    return knowledge_base_reply(user_text), "knowledge_base"

Best practices

  • Keep keys on the server only, loaded from environment variables, and rotate them regularly.
  • Trim the history: send only the last 6–10 messages. This cuts cost and keeps the context from bloating.
  • Usage limits: cap message length and the number of messages per session and per IP to prevent abuse and budget drain.
  • Monitoring: log the provider used, latency, tokens consumed, and fallback events. This is the data behind your next decision.
  • Short timeouts: 15–25 seconds for chat, then fall back. Users will not wait longer.
  • Regular comparison tests: run your fixed set of questions against every provider and compare quality and cost before changing the default.
  • Prompt injection: treat user input and retrieved documents as untrusted text, and never grant the model more permissions than it needs.

FAQ

Why not just use a single provider?

You can, but a unified layer reduces the risk of outages and price changes, and lets you compare and optimize without rewriting your application.

Don't want to manage all this yourself?

We build you a ready-made AI assistant that runs on the best model for each task and switches between them automatically to cut costs, without you touching a single line of code.

See the ready-made assistant