Postmortem: A Deep Dive into Cloudflare's November 2025 Outage

How a simple database permission change brought down 20% of the internet and what we can learn from it

The Day the Internet Held Its Breath

Picture this: you're trying to check ChatGPT for a quick answer, but instead you get an error page. You switch to Spotify for some music, another error. Twitter? Down. Discord? Nope. Even your favorite gaming site won't load.

Welcome to November 18, 2025, when a single configuration change at Cloudflare managed to break a significant chunk of the internet for nearly 6 hours. This wasn't some sophisticated cyberattack or natural disaster, it was something much more mundane and, frankly, scarier: a routine database permission update that went sideways.

What Actually Happened? (The TL;DR Version)

At 11:05 UTC, Cloudflare engineers made what seemed like a harmless change to their ClickHouse database permissions. The goal was simple: improve security by making database access more explicit and granular. What could go wrong?

Well, everything.

This innocent change caused their Bot Management system's configuration file to suddenly double in size. When this oversized file got pushed to Cloudflare's global network, it hit a hard-coded limit in their proxy software and caused it to panic. The result? HTTP 5xx errors for millions of websites that rely on Cloudflare's services.

The Scope: When 20% of the Internet Goes Dark

Let's put this in perspective. Cloudflare isn't just another tech company, it's basically the internet's immune system. They handle traffic for about 20% of all websites globally, with data centers in 330 cities and connections to 13,000 networks.

When Cloudflare goes down, it's not just a few websites that suffer. Major platforms that millions depend on daily suddenly become unreachable:

ChatGPT and OpenAI services - AI workflows ground to a halt
Spotify - No music streaming for millions
X (Twitter) - Social media conversations stopped
Discord - Gaming communities went silent
League of Legends - Gamers couldn't connect
Canva - Design work came to a standstill
Archive of Our Own - Fanfiction readers were left hanging

The irony? Cloudflare's main business is protecting websites from DDoS attacks and keeping them online. On this day, they became the very thing they're supposed to protect against.

The Technical Deep Dive: How a Database Query Broke the Internet

The ClickHouse Connection

To understand what went wrong, we need to talk about ClickHouse, the database system at the heart of this mess. ClickHouse is designed for analytics and handles massive amounts of data across multiple shards (think of them as separate database pieces that work together).

Cloudflare uses ClickHouse to power their Bot Management system, which generates "bot scores" for every request hitting their network. These scores help websites decide whether incoming traffic is from real humans or automated bots.

The Innocent Change That Started It All

The engineering team was working on improving security by making database permissions more explicit. Previously, users could only see tables in the "default" database when querying metadata. The change made it so users could also see metadata for underlying tables in the "r0" database.

Sounds reasonable, right? The problem was in this seemingly innocent SQL query:

SELECT name, type
FROM system.columns
WHERE table = 'http_requests_features'
ORDER BY name;

Notice what's missing? There's no filter for the database name. Before the change, this query would return columns from just the default database. After the change, it started returning duplicate entries, one from the default database and one from the r0 database.

The Cascade Effect

Here's where things get interesting (and terrifying). This query was used to generate the "feature file" for Cloudflare's Bot Management system. When the query started returning duplicates, the feature file suddenly doubled in size, going from about 60 features to over 200.

The Hard Limit That Broke Everything

Cloudflare's proxy system had a hard-coded limit of 200 features for performance reasons (memory preallocation). When the oversized file with 200+ features hit their servers, the system did what any well-behaved program should do when it encounters an unexpected condition: it panicked.

The specific error in their Rust code looked like this:

thread fl2_worker_thread panicked: called Result::unwrap() on an Err value

This panic translated directly into HTTP 5xx errors for end users. Every request that needed bot scoring (which is most of them) would fail.

The Intermittent Nature: Why It Kept Coming Back

One of the most confusing aspects of this outage was its intermittent nature. The internet would break, then mysteriously fix itself, then break again. This pattern initially led Cloudflare's team to suspect a DDoS attack.

The explanation is actually quite elegant in its simplicity. The problematic feature file was regenerated every 5 minutes by querying the ClickHouse cluster. Since Cloudflare was gradually rolling out the permission changes across their cluster, only some nodes would generate the bad file at any given time.

Every 5 minutes, there was essentially a coin flip: would the query run on an updated node (generating a bad file) or an old node (generating a good file)? Eventually, all nodes were updated and the system stabilized in the failing state.

The Human Element: When Coincidence Breeds Confusion

Adding to the chaos, Cloudflare's status page went down at the same time, even though it runs on completely separate infrastructure. This coincidence led the incident response team to initially suspect a coordinated attack targeting both their main systems and their communication channels.

Internal chat logs show the team's concern about potential "Aisuru DDoS attacks" - a reference to recent high-volume attacks they'd been dealing with. This false lead cost precious time during the early stages of the incident.

The Ripple Effects: Beyond Just Website Errors

Different Impacts for Different Systems

Not all Cloudflare customers experienced the same problems. The company was in the middle of migrating to a new proxy system (FL2), and the two systems failed differently:

FL2 customers: Saw HTTP 5xx errors (complete failures)
Legacy FL customers: Didn't see errors but got incorrect bot scores (all traffic scored as 0, leading to false positives for bot-blocking rules)

Downstream Service Chaos

The outage cascaded through Cloudflare's ecosystem:

Workers KV: Their key-value storage service failed, affecting applications built on it
Cloudflare Access: Authentication systems went down, locking users out
Turnstile: Their CAPTCHA alternative stopped working
Dashboard: Users couldn't log in to manage their Cloudflare settings
Email Security: Spam detection accuracy dropped

The Recovery: How They Fixed the Unfixable

Timeline of Recovery

The incident response followed a classic pattern of investigation, false starts, and eventual resolution:

11:32 - First detection and investigation begins 13:05 - Partial mitigation by bypassing core proxy for some services
14:24 - Root cause identified, bad file generation stopped 14:30 - Main impact resolved with deployment of good configuration file 17:06 - All services fully restored

The Fix

Once they identified the root cause, the fix was relatively straightforward:

Stop generating new Bot Management configuration files
Manually deploy a known-good version of the file
Restart the core proxy systems
Gradually bring dependent services back online

The technical fix was simple, but the investigation took hours because the intermittent nature of the problem made it incredibly difficult to diagnose.

What This Teaches Us About Internet Infrastructure

The Fragility of Centralization

This outage highlights a fundamental issue with modern internet infrastructure: we've created massive single points of failure. When companies like Cloudflare, AWS, or Google Cloud have problems, the effects ripple across the entire internet.

Think about it: one database query change at one company affected millions of websites and billions of users. That's both impressive and terrifying.

The Importance of Graceful Degradation

Cloudflare's system did exactly what it was designed to do when it encountered an error: fail fast and fail loudly. But there's a question worth asking: should a bot management system failure bring down the entire CDN?

Better systems might have:

Fallen back to a default bot score instead of panicking
Had circuit breakers to isolate failing components
Implemented gradual rollouts for configuration changes

The Testing Gap

This incident reveals a classic testing blind spot. The team tested their database permission changes, but they didn't test what would happen when those changes affected downstream systems that made assumptions about query results.

It's a reminder that in complex systems, the most dangerous bugs often live at the interfaces between components.

Lessons for the Rest of Us

For Developers

Question Your Assumptions: That SQL query worked fine for years until it didn't. Always consider what might change about your dependencies.
Implement Graceful Degradation: When a non-critical system fails, don't take down the whole application.
Test Integration Points: Unit tests are great, but integration tests catch the bugs that really hurt.
Monitor Everything: Cloudflare's monitoring caught this quickly, but better monitoring might have caught it before it hit production.

For System Architects

Avoid Hard Limits: That 200-feature limit seemed reasonable until it wasn't. Consider soft limits with graceful handling.
Design for Failure: Assume every component will fail and design accordingly.
Implement Circuit Breakers: Isolate failing components to prevent cascade failures.
Plan for Rollbacks: Have quick rollback mechanisms for configuration changes.

For Organizations

Diversify Dependencies: Don't put all your eggs in one CDN basket.
Have Incident Response Plans: Know how to communicate with users when things go wrong.
Practice Chaos Engineering: Regularly test how your systems handle failures.

The Bigger Picture: Internet Resilience in 2025

The Concentration Problem

This outage is part of a troubling pattern. In recent months, we've seen major outages at:

Amazon Web Services (affecting countless services)
Microsoft Azure (disrupting business operations globally)
Google Cloud (impacting everything from Gmail to YouTube)

Together, these three providers plus Cloudflare control a massive portion of internet infrastructure. When they fail, the internet fails.

The Path Forward

The solution isn't to abandon these services (they're incredibly valuable), but to build more resilience:

Regulatory Oversight: Should critical internet infrastructure be subject to more oversight?
Diversity Requirements: Should large organizations be required to use multiple providers?
Better Standards: Can we develop better standards for graceful degradation and failure isolation?
Investment in Alternatives: Supporting smaller, regional providers could reduce concentration risk.

What Cloudflare Is Doing About It

To their credit, Cloudflare has been transparent about this failure and outlined specific improvements:

Immediate Changes

Hardening Configuration Ingestion: Treating internal configuration files with the same validation as user input
Global Kill Switches: Better ability to quickly disable problematic features
Resource Protection: Preventing error reporting systems from overwhelming resources
Failure Mode Reviews: Systematic review of how each system component can fail

Cultural Changes

The incident has prompted deeper questions about testing, deployment practices, and system design philosophy. It's a reminder that even the most sophisticated systems can fail in unexpected ways.

The Human Cost

Beyond the technical details, it's worth remembering the human impact. Millions of people couldn't access services they depend on for work, entertainment, and communication. Small businesses lost revenue. Students couldn't access educational resources. The ripple effects of internet outages extend far beyond error pages.

Conclusion: Building a More Resilient Internet

The Cloudflare outage of November 2025 serves as a stark reminder of how interconnected and fragile our digital infrastructure has become. A single line of SQL code, missing a database filter, managed to disrupt the daily lives of millions of people worldwide.

But here's the thing: this kind of failure is inevitable in complex systems. The question isn't how to prevent all failures (that's impossible), but how to build systems that fail gracefully and recover quickly.

As we become increasingly dependent on digital infrastructure, we need to think seriously about resilience, diversity, and graceful degradation. The internet is too important to be this fragile.

The next time you can't access your favorite website, remember: somewhere, a developer is probably staring at a database query, wondering how something so simple could break something so big. And they're working as fast as they can to fix it.

Because at the end of the day, we're all just humans trying to keep the digital world running, one configuration file at a time.

What do you think? Have you experienced similar cascading failures in your own systems? How do you build resilience into your applications? Share your thoughts and war stories in the comments below.