Why AI Models Degrade Over Time: What Actually Happens, Dated

Key Takeaway

• The “it got worse” complaint is the industry’s most repeated one, and the record behind it is precise: models degrade over time on measured tasks — and often something else is going on. The four mechanisms, dated.

  🔁 Silent version swaps: the endpoint you call can become a different model without an announcement — with 268 tracked releases from 11 labs and a cadence roughly 4× the 2023 pace, drift is structural, not imaginary.

• 📊 The founding study: Stanford and Berkeley researchers documented GPT-4’s prime-number accuracy falling from 84 percent to 51 percent between March and June 2023 — and a code-metric drop of 42 points that vanished once the test was cleaned up.
• 🧪 Benchmarks decay, not just models: test questions leak into training data, so scores measure contamination as much as capability over time.
• 🪟 Context windows fill: long agent sessions thin out as the window fills — the sense that answers degrade over time within a session are often the session’s own memory budget running out.
• 🛠️ Disclosure arrived: OpenAI’s September 16, 2026 framework to track and disclose unsafe behavior — six examples included — is the industry’s formal answer to “models change without notice” — the documentation layer for the ways models degrade over time.

“It got worse” is the most repeated complaint about AI — and also the hardest to prove. Somewhere between the morning’s sharp answers and the afternoon’s confused ones, a model the user already trusted starts missing things it used to catch, and the user is left asking whether the model degraded, the question changed, or something in between. The clear answer arrived years ago in a research paper, and the infrastructure around models has changed enough since to make it current again: AI models can degrade over time on the tasks you measure — and most reports describe something slightly different from what the word assumes.

This guide lays out the four mechanisms behind AI model degradation reports — version churn, behavior drift, evaluation decay and context decay — each dated on the record, and the practical test for the week your model seems to degrade over time — or the morning it simply behaves differently.

The founding study: behavior changed under the same name

The paper that frames this whole topic is short and blunt: “How Is ChatGPT’s Behavior Changing Over Time?“, by Lingjiao Chen, Matei Zaharia and James Zou of Stanford University and UC Berkeley. The design: take the March 2023 and June 2023 versions of GPT-4 and GPT-3.5, run identical tests on both, measure the drift. The results, published in the MIT Harvard Data Science Review’s Spring 2024 issue, are the anchor every later discussion still cites.

The drift was material and it was uneven. GPT-4’s accuracy at identifying prime numbers fell from 84 percent in March to 51 percent in June; the same June GPT-4 stopped following the chain-of-thought instruction format that the test assumed, and its responses shortened dramatically. Counting tasks showed the same pattern: 83 percent falling to 35 percent. Meanwhile GPT-3.5 moved the opposite way — its prime-identification rose and its answers grew longer. The authors’ own framing is the careful one: the models’ behavior “varied significantly over a relatively short amount of time,” which is not the same as saying everything got worse everywhere. Some abilities improved while others regressed, in the same three-month window, behind the same product name.

The study’s second finding caused the argument that followed. A code benchmark scored the June GPT-4 forty-two points worse on directly executable code — but the drop was partly a formatting choice: the newer model wrapped code in explanatory text, so a naive “runs as-is” test counted helpfulness as failure. Ars Technica’s coverage quoted the critics making exactly this point, and when the same study re-scored after stripping non-code text, the June GPT-4’s result improved to 70 percent — higher than March. Two defensible readings, opposite headlines, and the whole industry’s degradation debate inside one example.

Why the problem has grown since: version churn at 4× pace

The March-versus-June study caught its drift because the researchers happened to snapshot twice. Today’s release cycle makes snapshots a lifestyle requirement. The AI Release Tracker counts 268 frontier models from 11 companies since late 2022, and its data — cited by Business Insider — shows the monthly cadence of major releases has roughly quadrupled since 2023. Google shipped Gemini 4 Argon last week and has its Carbon build in development; OpenAI is preparing its Terra family; Anthropic tests a reasoning-focused model that infers goals from limited instructions, per The Information’s November report. The naming changes alone read like a weather system.

Every one of those releases can move behavior on any task you care about, and the API changelog is the only formal notice most users ever see. That is the first mechanism behind the “models degrade over time” reports: not a model rotting, but an endpoint quietly becoming a different model. The fix is behavioral: version-pin where the platform allows it, snapshot the outputs that matter, and re-run your own short battery after any release — the same test the researchers ran, sized for one workflow (the tracker tools we use live in our AI-browser field guide).

Evaluation decay: when the test stops measuring

The second mechanism runs the opposite direction: the model stays capable and the test gets easier. Public benchmarks leak into training data — the questions, the answers, and sometimes entire answer keys — so later releases train directly on the exam. Scores then rise without the underlying ability moving, which is why top-leaderboard numbers increasingly say less than they appear to: GPQA Diamond’s leading score and SWE-Bench Verified’s leader are quoted on trackers that now must rotate or gate questions to keep the measurement honest.

The practical effect on the user is subtle: a model whose score went up can still feel like it started to degrade over time on your task, because your task was never the benchmark. The score the user trusts is their own battery — the same prompts, the same checks, run on a schedule. For content and data teams, that battery is now standard tooling: the short script that runs ten real prompts nightly and compares outputs is the difference between noticing drift in a week and reading about it in a quarterly report — the same hygiene our enterprise agents guide applied to connector governance.

Context decay: the session, not the model

The third mechanism needs no release at all: every meeting has a memory budget, and the session spends it. Long agent runs fill their context windows, instructions drift out of reach, earlier decisions get forgotten mid-task, and the answers decline while the model itself is byte-for-byte unchanged — the session, not the update, makes it seem to degrade over time. Reload the conversation fresh and the “degradation” is gone, which is exactly the symptom pattern: gets worse within a session, returns after a break.

Usage limits produce the same illusion by another door. When a request hits its cap, the response can thin out, truncate or fall back — the visible sign looks like a dumber model when it is actually a smaller slice of one. The practical counter is to restart sessions at known intervals, keep the standing instructions near the front of the window, and split long workflows into stages so no single session carries the whole job’s memory. The teams that feel the least sense that their models degrade over time are usually the ones with the cleanest session hygiene.

Formal answers to the “changed without notice” problem

The industry has started building its own record. OpenAI introduced a framework on September 16, 2026, for tracking and disclosing instances of its technology showing unsafe behavior — and published six examples with it, an acknowledgment that behavior can change and a mechanism for saying so in public. Version-pinning, changelogs and system-cards are spreading from research papers into product surfaces; the trackers that count releases now also track what changed. None of this makes drift vanish. It makes drift accountable — a logged change a builder can react to, instead of a Tuesday where the tool is different and no one said so.

The model-degradation debate has also settled into a cleaner set of questions. Capability regression is now checked against held-out tests rather than public benchmarks; behavior change is measured on task-level batteries rather than vibes; and “it got worse” reports carry a burden of a pinned version and a fixed prompt before they mean anything. The founding study’s last line said it first: because it is not transparent how these models are updated over time, continuous evaluation is the user’s job — the same conclusion the release numbers make obvious today, and why users keep saying models degrade over time: models can degrade over time without anyone announcing it.

How to test whether your model degrades over time — in one morning

The reading test that separates the four mechanisms fits a single sitting. First, pin and record: save the model version, date, temperature settings and the exact prompts you care about. Second, snapshot: run your battery and keep the outputs. Third, separate session from model: rerun the same prompts in a fresh session with a re-loaded context — if the quality returns, the problem was context decay, not the model. Fourth, check the changelog: if a new version shipped in your window, read the notes — a behavior change on your task may be a documented one. Fifth, re-score against a private battery rather than a public benchmark, so evaluation decay cannot touch the result. If a genuine regression survives all five steps, you have a degradation case worth reporting — with the two artifacts that make it arguable: the snapshot and the version.

What model degradation means for teams in the Philippines and beyond

For Philippine teams running client work on these platforms, the degradation lesson is operational, not philosophical. The BPO and shared-services sector’s AI-assisted workflows — document review, data pulls, report drafting — inherit every version change the provider ships, at Philippine office hours whether or not anyone announced it. The teams that treat their battery of check prompts as part of quality control are the ones who notice drift before the client does; the nightly re-run takes minutes and saves the conversation.

For the developers and freelancers building products on these APIs, version-pinning is the difference between a reproducible product and a Tuesday surprise: the pin costs nothing, the drift costs everything — and the drift arrives through the same governed windows our managed MCP servers guide covers. And for the broader professional audience the standard advice has not changed since the founding study — trust your recorded outputs more than your impression, and re-baseline when the provider admits what changed. The record shows the models move; the professionals who keep their own record are the ones the movement does not surprise.

FAQ

Why do AI models seem to get worse — do they degrade over time?

Four mechanisms explain why people say models degrade over time: version churn (the endpoint silently becomes a different model), behavior drift (updates trade one ability for another), evaluation decay (benchmark questions leak into training, so scores stop measuring real ability) and context decay (long sessions fill their memory budget). The founding evidence came from the Stanford and Berkeley study that measured GPT-4’s prime-number accuracy falling from 84 to 51 percent between March and June 2023, and found the “degradation” was often a format change counted as a failure.

What did the Stanford Berkeley study on ChatGPT actually find?

Chen, Zaharia and Zou compared the March 2023 and June 2023 versions of GPT-4 and GPT-3.5 on identical tasks, published through the MIT Harvard Data Science Review. GPT-4’s behavior changed materially in three months — prime identification 84 to 51 percent, a counting task 83 to 35 percent, a large drop in directly executable code and a dramatic shortening of answers — while GPT-3.5 moved in the opposite direction on the same tasks. Their conclusion: behavior varies over short periods, so applications need continuous evaluation, especially because model updates are not transparently announced.

Is ChatGPT worse now than when it launched?

Different, and measured rather than judged: the tracked releases show capability moving on individual tasks in both directions, while benchmarks themselves have changed reliability as test questions have leaked into training data. The clean way to answer the question for your own work is a private battery — the same prompts run on a schedule, outputs saved, changes counted. An impression without a version pin and a fixed prompt cannot separate model change from context decay or the test itself.

Why does my AI assistant get worse during a long session?

Long sessions fill the context window the model uses as its working memory, so earlier instructions drift out of reach and answers thin out — while the model is unchanged. Usage limits can thin responses the same way. The test is instant: start a fresh session with the key instructions re-loaded. If the quality returns, the session was the problem; restart at intervals and keep standing instructions at the front of the window.

What is evaluation decay, and why do I keep hearing benchmarks are unreliable?

Evaluation decay is the erosion of a benchmark’s meaning: when its questions and answers leak into training data, later models score high on it without gaining ability. That is why tracker sites now rotate or gate benchmark questions, and why a public leaderboard score that climbs tells you less than it seems to. A private test set that never leaves your organization stays honest.

How can I check my own model for degradation?

Pin and record the version, date and settings; snapshot the outputs of a short battery of real prompts; rerun in a fresh session to rule out context decay; check the provider’s changelog for a new version; and re-score against a private battery. If a genuine regression survives those steps, you have a documented case — the snapshot and the version are the two artifacts that make a degradation report worth reading.

Financial Disclaimer

This article is published for general information and analysis, not as investment advice, and it names no security for purchase. Model platforms, pricing and capabilities change without notice, and figures cited reflect public reporting as of their dates. Assess your own requirements before acting on the guidance here; WorldNgayon accepts no liability for decisions taken on the basis of this article.

@online{agron2026whyai,
  author       = {Agron, Edmon},
  title        = {{Why AI Models Degrade Over Time: What Actually Happens, Dated}},
  year         = {2026},
  date         = {2026-11-07},
  organization = {WorldNgayon},
  url          = {https://worldngayon.com/why-ai-models-degrade-over-time/},
  urldate      = {}
}

TY  - ELEC
AU  - Agron, Edmon
TI  - Why AI Models Degrade Over Time: What Actually Happens, Dated
T2  - WorldNgayon
PY  - 2026
DA  - 2026/11/07/
UR  - https://worldngayon.com/why-ai-models-degrade-over-time/
Y2  - 
ER  -