Ultrahost/ docs
All examples

Example 10

Experiment

AI Model Gateway

One front door for every model call, with memory, limits, and a bill you can read.

Calling a model directly is easy and expensive. A gateway chooses the model, remembers what was asked, reuses answers that are close enough, measures what it costs, and controls who may use what.

Chapter one

When a request comes in

Every call goes through the same front door. The gateway checks who is asking, looks for an answer it already has, and only pays for a model when it has to.

  1. 1
    Authorization Layer

    The caller is checked

    The Authorization engine determines who is allowed to use which LLMs, before anything is spent.

  2. 2
    Vector DB

    Similar past requests are looked up

    The Vector engine finds earlier inputs that mean nearly the same thing, so their responses can be reused.

  3. 3
    File Server

    A cached answer is served

    Only when a close enough match exists

    The File engine holds a long-term cache of answers, so a close enough match comes back without a model call.

  4. 4
    LLM Plane

    A model is chosen and called

    Only when nothing can be reused

    The LLM engine finds an available model that fits the budget, the complexity, and the response time the request needs, and calls it.

  5. 5
    File Server

    The new answer joins the cache

    Only after a new model call

    The fresh response is stored in the long-term cache, so the next similar request does not need a model at all.

Chapter two

When the call is remembered

A gateway is only useful if you can see what went through it. Three blocks keep a record you can read, measure, and search.

Every call leaves a trace

Whether it hit a model or the cache, each request is worth keeping a record of.

  • What happened

    The run is recorded

    The Data engine keeps the input and the output of every LLM run, so any answer can be traced back to what was asked.

    Document Database

  • What it cost

    Usage becomes a bill you can read

    The Timeseries engine tracks tokens, cost, latency, and more over time, so you can see what each model and each team is spending.

    Timeseries Database

  • What was said

    Past answers are easy to find

    The Search engine indexes the inputs and outputs, so finding a response from the past takes seconds instead of a rerun.

    Search Engine

Make it yours

Variations worth trying

  • Start by only logging calls and their cost, and add caching once you see the repeats.
  • Cache exact matches first, and add similarity when exact matches are rare.
  • Give each team its own model allowance instead of one shared list.

Starting point

Ask your coding agent for your version

Use this prompt as a starting point, then change whatever does not fit your idea.

Build an AI model gateway that routes calls to available models, tracks tokens, cost, and latency, caches and reuses similar answers, and controls who can use which models.

Use these Ultrahost building blocks, in this flow.

When a request comes in:
1. Authorization Layer: The Authorization engine determines who is allowed to use which LLMs, before anything is spent.
2. Vector DB: The Vector engine finds earlier inputs that mean nearly the same thing, so their responses can be reused.
3. File Server (only when a close enough match exists): The File engine holds a long-term cache of answers, so a close enough match comes back without a model call.
4. LLM Plane (only when nothing can be reused): The LLM engine finds an available model that fits the budget, the complexity, and the response time the request needs, and calls it.
5. File Server (only after a new model call): The fresh response is stored in the long-term cache, so the next similar request does not need a model at all.

When the call is remembered:
- What happened (Document Database): The Data engine keeps the input and the output of every LLM run, so any answer can be traced back to what was asked.
- What it cost (Timeseries Database): The Timeseries engine tracks tokens, cost, latency, and more over time, so you can see what each model and each team is spending.
- What was said (Search Engine): The Search engine indexes the inputs and outputs, so finding a response from the past takes seconds instead of a rerun.

Treat this as a starting point. Keep the parts that fit my idea and adapt the rest.

Next example

AI Knowledge Base