Unthrottled Cloud Serverless Runaways
The Recursive Function That Burned Thousands Over a Single Weekend
The Incident Post-Mortem
In early 2023, we deployed a serverless event worker designed to process uploaded user documents. The function was configured to listen for new files in an Amazon S3 storage bucket, parse the document text, and write an audit log back into the same storage bucket.
A developer made a minor configuration mistake: they placed the output audit log in the root directory rather than an isolated subfolder. When the function wrote the first audit log, S3 emitted a new file creation event, which triggered the function again, which wrote another log, which triggered another event.
By Friday midnight, the function was executing in an exponential recursive loop, spinning up thousands of concurrent serverless instances that ran millions of times per hour. Because our cloud billing alert was configured to evaluate only at monthly thresholds, nobody received a notification until Sunday morning.
By the time we terminated the function, AWS had billed $8,400 for 42 million unnecessary invocations. We immediately implemented hard concurrency throttles on all serverless workers, isolated event loops, and deployed real-time hourly spend alarms with automated circuit breakers.
Root Cause Technical Analysis
Event-driven serverless function triggering an infinite recursive loop on S3 bucket writes. In complex engineering systems, superficial fixes frequently obscure the underlying physics of software execution. Doubling server resources or adopting distributed patterns without verifying basic IO patterns simply defers the inevitable failure while increasing cost.
The Permanent Operational Rule
Rule 09: Every serverless function must have explicit concurrency caps, hard timeout limits, and real-time billing alarms.
How We Prevent This Incident Today
Today, every production service deployed across our ventures adheres to automated verification gates that prevent this failure mode. We run automated query budget linters, enforce deterministic cryptographic schemas, cap serverless concurrency pools, and maintain strict hardware requirements before deploying customer-facing systems.
Book a 1-on-1 Systems Architecture Review
INR 2,000 / $20 USDSchedule a 45-minute private consultation with Arkajit Das. Review your technical architecture, prevent costly infrastructure mistakes, and optimize your systems.