The Cloudflare Outage That Broke 28% of the Internet

So here we are again. Another day, another major outage that reminds us just how fragile our digital infrastructure really is. On December 5, 2025, Cloudflare managed to take down a chunk of the internet for about 25 minutes, affecting roughly 28% of their HTTP traffic. And honestly? The way it happened is both fascinating and terrifying.

Let me walk you through what went down, why it matters, and what we can all learn from this mess.

The Perfect Storm: When Security Fixes Become Security Risks

Here's the thing that gets me about this whole situation. Cloudflare wasn't being reckless or pushing some experimental feature. They were literally trying to protect their customers from a critical React Server Components vulnerability (CVE-2025-55182). You know, doing the right thing.

But as we've all learned the hard way, the road to production hell is paved with good intentions.

What Actually Broke?

The technical details here are pretty wild. Cloudflare's Web Application Firewall (WAF) normally buffers HTTP request bodies up to 128KB for analysis. To better protect against the React vulnerability, they decided to bump this up to 1MB, which matches Next.js defaults.

Sounds reasonable, right? Well, here's where things get interesting.

The first change (buffer size increase) was being rolled out gradually. Smart move. But then their internal testing tool couldn't handle the bigger buffer size. No big deal, they thought, let's just turn it off.

Here's where everything went sideways. Instead of using their gradual deployment system, they used their global configuration system to disable the testing tool. This system pushes changes to their entire network within seconds.

The Bug That Lived in the Shadows for Years

Now we get to the really juicy part. The actual bug that caused the outage had been lurking in their codebase for years, just waiting for the right conditions to strike.

When Cloudflare's killswitch system disabled the testing rule, it correctly skipped the "execute" action. But then this piece of Lua code tried to process the results:

if rule_result.action == "execute" then
  rule_result.execute.results = ruleset_results[tonumber(rule_result.execute.results_index)]
end

The problem? The code assumed that if a rule has action="execute", the rule_result.execute object would exist. But since the rule was skipped, that object was nil. Boom. Runtime error.

This is exactly the kind of bug that makes you go "how did this not get caught earlier?" It's a classic null pointer dereference, something that languages with strong type systems would catch at compile time.

Why Didn't This Happen Before?

Here's what's fascinating: Cloudflare had used their killswitch system many times before without issues. The difference? They'd never applied it to a rule with an "execute" action. It was a perfect storm of circumstances that finally triggered this dormant bug.

The Cascading Failure Pattern

Let's talk about what makes this outage particularly interesting from a systems perspective. It wasn't just one thing that went wrong, it was a cascade of seemingly reasonable decisions that combined into a disaster.

Each individual decision made sense in isolation:

• Protecting customers from a security vulnerability? Good.
• Increasing buffer size to match industry standards? Reasonable.
• Disabling a broken internal tool? Logical.
• Using the global config system for a quick fix? Well...

The Real Problem: Configuration as Code vs Configuration as Chaos

Here's where I think Cloudflare's real issue lies. They have two different deployment systems with completely different risk profiles:

1. Gradual deployment system: Rolls out changes slowly, allows for monitoring and rollback
2. Global configuration system: Pushes changes to the entire network in seconds

Having these two systems with such different behaviors is like having a sports car and a tank in your garage, but no clear rules about when to use which one.

The fact that they used the global system for what seemed like a "harmless" internal tool change shows a fundamental problem with their change management process.

But Wait, There's More Context

This wasn't Cloudflare's first rodeo with global outages. Just two weeks earlier, on November 18, they had a similar incident caused by a global configuration change. They even acknowledged that they were working on fixes to prevent exactly this type of problem.

So why did it happen again? Because those fixes weren't deployed yet.

What This Means for the Rest of Us

Okay, so Cloudflare messed up. But what can we learn from this? A lot, actually.

Lesson 1: Type Safety Isn't Just Academic BS

That Lua bug that caused the outage? It's exactly the kind of thing that Rust's type system would catch at compile time. Cloudflare even mentions that their new FL2 proxy, written in Rust, didn't have this issue.

This isn't about Rust vs Lua specifically. It's about choosing tools that help you catch errors before they reach production. Whether that's TypeScript instead of JavaScript, or just better testing practices.

Lesson 2: Configuration Changes Are Code Changes

Too many organizations treat configuration changes as "safe" compared to code changes. But as this outage shows, a configuration change can be just as dangerous as any code deployment.

Your configuration changes should go through the same rigor as your code changes:

• Code review
• Testing in staging environments
• Gradual rollouts
• Monitoring and rollback capabilities

Lesson 3: The Blast Radius Problem

When you have systems that can affect your entire infrastructure instantly, you need to be extra careful about how you use them. Cloudflare's global configuration system is powerful, but with great power comes great responsibility (and great potential for spectacular failures).

Lesson 4: Security vs Reliability Trade-offs

This outage happened because Cloudflare was trying to protect their customers from a security vulnerability. It's a perfect example of how security and reliability concerns can conflict.

The question isn't whether to prioritize security or reliability, it's how to implement security improvements without compromising reliability. In this case, a more gradual approach to the entire change (including the testing tool modification) might have prevented the outage.

The Bigger Picture: Internet Infrastructure Fragility

Let's zoom out for a second. Cloudflare handles a massive portion of internet traffic. When they go down, a significant chunk of the web goes with them. This raises some uncomfortable questions about internet infrastructure centralization.

We've seen similar issues with:

• AWS outages taking down huge portions of the internet
• Facebook/Meta outages affecting billions of users
• DNS provider outages making websites unreachable

The internet was designed to be decentralized and resilient, but in practice, we've created these massive single points of failure.

What Cloudflare Is Doing About It

To their credit, Cloudflare is being pretty transparent about their plans to prevent these kinds of incidents. They've outlined several initiatives:

Enhanced Rollouts & Versioning

They want to apply the same gradual deployment practices they use for software to their configuration changes. This should help prevent global impacts from configuration bugs.

Streamlined Break Glass Capabilities

Making sure they can still perform critical operations even when their systems are partially broken. This is crucial for incident response.

"Fail-Open" Error Handling

Instead of failing hard when something goes wrong, their systems will try to degrade gracefully. This is a huge shift in philosophy and could prevent many types of outages.

But here's the thing that bothers me: they said they were working on these improvements after the November 18 outage, but they weren't ready when this December 5 outage happened. That's a pretty short window, but it shows how quickly these issues can compound.

The Human Element

One thing that often gets overlooked in these post-mortems is the human element. Imagine being the engineer who made that configuration change. You're trying to protect customers from a security vulnerability, you notice an internal tool is broken, you make what seems like a harmless change to disable it, and suddenly you've taken down 28% of Cloudflare's traffic.

That's got to be a terrible feeling. And it highlights why we need systems and processes that protect us from these kinds of mistakes, because humans are going to make them.

Questions This Raises

This outage brings up some interesting questions that don't have easy answers:

How do you balance security response speed with deployment safety? When a critical vulnerability is disclosed, there's pressure to fix it quickly. But rushing can lead to exactly these kinds of problems.

Should critical infrastructure providers have different standards? Cloudflare isn't just any company, they're part of the internet's backbone. Should they be held to higher standards for change management?

How do you test for edge cases that happen once in years? That Lua bug existed for years without being triggered. How do you test for scenarios that rare?

Is it better to fail fast or fail gracefully? Cloudflare's system failed fast and hard, making the problem obvious but also maximizing impact. Would a more gradual failure have been better?

The Technical Debt Reality

Here's something that really stands out to me: this bug existed in their FL1 proxy but not in their newer FL2 proxy written in Rust. This suggests they knew about the limitations of their older system but hadn't fully migrated away from it yet.

This is technical debt in action. You know you have better, safer systems, but you can't just flip a switch and migrate everything overnight. In the meantime, you're stuck maintaining and running systems you know have problems.

It's a reminder that technical debt isn't just about code quality or developer productivity. It can directly impact reliability and customer experience.

What You Can Do

If you're running your own systems, here are some concrete takeaways:

Audit Your Deployment Systems

Do you have multiple ways to deploy changes with different risk profiles? Make sure everyone on your team understands when to use which system.

Implement Gradual Rollouts for Everything

Not just code changes. Configuration changes, feature flags, even infrastructure changes should be rolled out gradually when possible.

Invest in Type Safety

Whether it's using TypeScript, Rust, or just better testing, invest in tools and practices that catch errors before they reach production.

Practice Incident Response

Have you actually tested your rollback procedures? Do they work when you're under pressure at 3 AM?

Monitor Configuration Changes

Treat configuration changes with the same monitoring and alerting rigor as code deployments.

The Silver Lining

Despite the frustration and impact, there are some positive aspects to how this played out:

1. Quick Detection and Response: The issue was identified and resolved in about 25 minutes
2. Transparency: Cloudflare published a detailed post-mortem quickly
3. Learning: They're using this as an opportunity to improve their systems
4. No Security Breach: Unlike some outages, this wasn't caused by malicious activity

Looking Forward

The reality is that outages like this are going to keep happening. The internet is incredibly complex, and we're constantly pushing the boundaries of what these systems can handle. The goal isn't to eliminate all outages (that's impossible), but to minimize their frequency and impact.

What gives me hope is that companies like Cloudflare are being transparent about their failures and investing in better systems. The move from Lua to Rust for their proxy, the work on gradual configuration rollouts, and the focus on fail-open error handling are all steps in the right direction.

But it's also a reminder that we can't take internet reliability for granted. Every time you load a webpage instantly, send a message, or stream a video, there are thousands of systems working perfectly in the background. When they don't work perfectly, we all feel it.

The Bottom Line

The December 5 Cloudflare outage is a perfect case study in how complex systems fail. It wasn't caused by a single bad decision or a catastrophic bug. It was the result of multiple reasonable decisions that combined in an unexpected way to trigger a dormant issue.

The technical details are fascinating, but the real lessons are about process, culture, and the inherent challenges of running critical internet infrastructure. As our digital world becomes more interconnected and dependent on a few key providers, these kinds of incidents become more impactful.

The good news? We're getting better at learning from these failures. The bad news? There will definitely be more of them.

So the next time your favorite website is down, remember that somewhere, there's probably an engineer having a very bad day, trying to figure out how a simple configuration change managed to break the internet. Again.

What do you think? Have you experienced similar cascading failures in your own systems? How do you balance the need for quick security fixes with deployment safety? Let me know in the comments.