Short version: A hosted model endpoint is not a fixed thing. This week DeepSeek proved it — V4-Flash-0731 is a full retrain shipped under the same name and the same deepseek-v4-flash endpoint, with zero migration and zero notice. If you call it, your production behavior may have moved on July 31 and nothing in your code, logs, or the API's version field would tell you. Here's a three-layer tripwire that costs almost nothing and catches the swap before a user does.

The failure mode#

You tuned a prompt against a model. It worked. Months of traffic flow through it. Then the provider retrains the model — for real, benchmark-beating reasons — and redeploys it behind the same alias you've been calling. Your parser was tuned to the old output format; your few-shot examples were shaped to the old quirks; your refusal handling assumed the old boundary. None of that is guaranteed to survive a retrain, and you got no version bump to prompt a re-test.

This isn't a DeepSeek problem; it's a hosted-endpoint property. The defense is to stop trusting the endpoint as your contract and make your own eval the contract instead.

Layer 1 — Pin what you can#

Where a provider publishes dated snapshot aliases — the -2026-07-31-style suffixes OpenAI and Anthropic offer — call the dated string, not the floating name. The floating name (gpt-..., deepseek-v4-flash) is defined to be the one that moves.

Then log the exact model string you sent as a field on every request:

resp = client.chat.completions.create(model=MODEL_ID, messages=msgs, seed=7, temperature=0)
log.info("llm_call", model_sent=MODEL_ID,
         system_fingerprint=getattr(resp, "system_fingerprint", None))

That system_fingerprint (when a provider returns one) is a weak hint the backend changed — treat it as a hint, not proof. The hard limit of pinning: not every model exposes a dated snapshot. DeepSeek's flash line doesn't, so pinning alone can't save you here. That's why Layer 2 exists.

Layer 2 — A canary prompt set (the ground truth)#

Keep 15–40 frozen prompts that exercise the behaviors you actually depend on: a required output format, a known edge case, a refusal boundary, a representative tool call. Run them on a schedule (a cron job or a CI step), at temperature 0 with a fixed seed where the API supports one, and hash the outputs:

import hashlib, json
def canary_hash(client, prompts):
    outs = []
    for p in prompts:
        r = client.chat.completions.create(
            model=MODEL_ID, messages=p, temperature=0, seed=7, max_tokens=512)
        outs.append(r.choices[0].message.content)
    return hashlib.sha256(json.dumps(outs).encode()).hexdigest()

today = canary_hash(client, CANARY_PROMPTS)
if today != LAST_KNOWN_HASH:
    alert("Canary hash changed — the model behind %s may have moved" % MODEL_ID)

Identical inputs producing different outputs is the definition of the event you're trying to catch — and it fires regardless of what the version field says. Some nondeterminism survives even at temperature 0, so confirm a change by re-running before you page anyone; a persistent flip is a real swap.

Do not rely on the returned version number instead. Providers retrain behind stable identifiers on purpose; the same-name retrain is precisely the case that a version-only check waves through.

Layer 3 — Alarm on behavior in production#

Your canary set can't cover everything real traffic does, so watch the distributions you already emit: output length, refusal rate, latency, and tool-call shape. A step change in any of them — average completion suddenly 20% longer, refusals doubling, a tool-arg schema drifting — is an early warning on live traffic that something moved. It's noisier than the canary (real traffic drifts too), which is why it's the alarm, not the verdict.

When the tripwire fires#

  1. Freeze to a pinned dated snapshot if one exists, buying time on the old behavior.
  2. Run your golden eval set — the real one, not the canary — against the new model.
  3. Decide. A silent upgrade is often a genuine improvement. But you make the adopt-or-roll-back call deliberately, against your own metrics, on your schedule — not the provider's deploy calendar. Our shadow-vs-canary-vs-A/B rollout guide covers the promotion path once you've chosen to adopt.

That's the whole discipline, and it's the same one behind pinning your judge model and pinning your agent stack: the endpoint is not the contract. Your eval is.