About • Failure Museum • Exhibit #09

Unthrottled Cloud Serverless Runaways

The Recursive Function That Burned Thousands Over a Single Weekend

Operating Era
2023 - Cloud Architecture and FinOps
Cost / Damage
$8,400 Cloud Invoice Over a 48-Hour Weekend
Documented Status
Resolved • Rule Enforced
Rogue Invocations
42,000,000 executions
Weekend Cloud Bill
$8,400 unexpected cost
Loop Duration
38 hours unmonitored
New Safeguard
Hourly circuit-breaker spend limits

The Incident Post-Mortem

In early 2023, we deployed a serverless event worker designed to process uploaded user documents. The function was configured to listen for new files in an Amazon S3 storage bucket, parse the document text, and write an audit log back into the same storage bucket.

A developer made a minor configuration mistake: they placed the output audit log in the root directory rather than an isolated subfolder. When the function wrote the first audit log, S3 emitted a new file creation event, which triggered the function again, which wrote another log, which triggered another event.

By Friday midnight, the function was executing in an exponential recursive loop, spinning up thousands of concurrent serverless instances that ran millions of times per hour. Because our cloud billing alert was configured to evaluate only at monthly thresholds, nobody received a notification until Sunday morning.

By the time we terminated the function, AWS had billed $8,400 for 42 million unnecessary invocations. We immediately implemented hard concurrency throttles on all serverless workers, isolated event loops, and deployed real-time hourly spend alarms with automated circuit breakers.

Root Cause Technical Analysis

Event-driven serverless function triggering an infinite recursive loop on S3 bucket writes. In complex engineering systems, superficial fixes frequently obscure the underlying physics of software execution. Doubling server resources or adopting distributed patterns without verifying basic IO patterns simply defers the inevitable failure while increasing cost.

The Permanent Operational Rule

Permanent Law

Rule 09: Every serverless function must have explicit concurrency caps, hard timeout limits, and real-time billing alarms.

How We Prevent This Incident Today

Today, every production service deployed across our ventures adheres to automated verification gates that prevent this failure mode. We run automated query budget linters, enforce deterministic cryptographic schemas, cap serverless concurrency pools, and maintain strict hardware requirements before deploying customer-facing systems.

Related Tools & Operating Playbooks
Cloud Savings Calculator →Latency Simulator → All 10 Museum Exhibits →

Book a 1-on-1 Systems Architecture Review

INR 2,000 / $20 USD

Schedule a 45-minute private consultation with Arkajit Das. Review your technical architecture, prevent costly infrastructure mistakes, and optimize your systems.

Book Calendar Slot → PayPal $20 USD → Back to Failure Museum →