Skip to main content
← All use cases
drop_in_proxy.pymemory_basics.py

​
The scenario

Memory makes every prompt bigger. That is the deal — and it is fine until the bill arrives and nobody can say how much of it was memory, or whether the tenth recalled item was worth anything. Two levers, and both are measurable rather than vibes.

​
Step 1 — Put a hard ceiling on the block

get_context returns a prompt-ready string with the budget enforced server-side. Whole memories are dropped, lowest-ranked first — never text cut mid-sentence, which would leave the model reading half a fact. This beats a client-side loop because your loop does not know the ranking.

​
Step 2 — Make the block cacheable

Providers discount an unchanged prompt prefix. A memory block reordered by relevance every turn destroys that prefix on every turn.
stable orders by when memories were created rather than by score, so the block mostly does not change between turns and the discount can apply.

​
Step 3 — Read what it actually cost

Dashboard → Usage → Memory in your prompts. It appears once the proxy has carried a turn, and shows three readings for the last 30 days:
  • Context injected (context_tokens) — what memory added to your prompts.
  • Cache-eligible (cache_prefix_tokens) — how much of that was unchanged since the previous turn: the measured eligibility for a provider discount, not an estimate of savings.
  • Stable share — the second as a percentage of the first.
These are dashboard figures. client.get_usage() is the API-key view of your credits (credits_remaining, credits_limit, credits_used, rate_limit_per_min) and does not carry them.

​
Evals

  1. Run twenty representative turns with max_tokens unset. Record Context injected from the Usage page.
  2. Re-run with a ceiling at roughly half. Score answer quality on the same questions.
  3. Find the point where quality drops. That number is your budget; most spaces sit well below the default.
  4. Run the same twenty turns in relevance and then stable order and compare Cache-eligible after each. Relevance mode should read near zero — that contrast is the evidence the setting is doing anything.

​
Guardrails

cache_prefix_tokens reports eligibility, not money saved. Whether a discount applies is the provider’s business, and it depends on their cache state. Do not present this figure to a customer as a saving.
  • Stable ordering trades freshness for cacheability. The most relevant memory is no longer first in the block. If answer quality drops, this is the first thing to turn off.
  • Budget selection happens in relevance order, then emission is reordered — so a tighter budget drops the weakest memory, not the newest one.
  • There is no baseline to compare against. What you would have spent without a memory layer is unobservable, and any “X% saved” number built on a guessed baseline will not survive being asked how it was calculated.

​
Built from

Both are complete, runnable scripts: drop_in_proxy.py, memory_basics.py.