About • Failure Museum • Exhibit #01

Over-provisioned Database Clusters

Believing Vertical Scaling Was Cheaper Than Fixing N+1 Queries

Operating Era
2017 - Healthcare Data Warehousing
Cost / Damage
$42,000 Surplus Infrastructure Waste
Documented Status
Resolved • Rule Enforced
Peak Instance Cost
$14,000 / month
Query Round-Trips
8,000 per request
Optimized Latency
85 milliseconds
Core Reduction
64 cores down to 16 cores

The Incident Post-Mortem

In early 2017, while scaling healthcare analytics pipelines for regional hospital reporting, dashboard query latencies spiked from 250 milliseconds to over 12 seconds during morning shift handoffs. Instead of inspecting SQL query execution plans, our team made the classic junior mistake: we doubled the virtual database instance size from 16 cores to 32 cores, and then to 64 cores with 256 gigabytes of RAM.

The immediate symptoms subsided for two weeks, but monthly cloud invoices surged by $42,000 over the subsequent quarter. When morning concurrency doubled again, the expensive database froze completely.

Upon conducting a rigorous slow-query trace, we discovered an unindexed N+1 query loop inside the patient discharge view. For every ward record displayed, our application fired 40 secondary queries to retrieve bed assignment logs. A single page load was executing 8,000 database round-trips.

We rewrote the query using an indexed SQL join and added a 60-second in-memory cache for static ward metadata. Dashboard latency plummeted to 85 milliseconds. We immediately downscaled the database back to 16 cores. That painful $42,000 lesson laid the groundwork for the FinOps principles that later enabled our $12 million cloud cost reduction at CAI.

Root Cause Technical Analysis

Unindexed relational joins generating 8,000 redundant SQL queries per page load. In complex engineering systems, superficial fixes frequently obscure the underlying physics of software execution. Doubling server resources or adopting distributed patterns without verifying basic IO patterns simply defers the inevitable failure while increasing cost.

The Permanent Operational Rule

Permanent Law

Rule 01: Profile and optimize query execution plans before upgrading database CPU or RAM tiers.

How We Prevent This Incident Today

Today, every production service deployed across our ventures adheres to automated verification gates that prevent this failure mode. We run automated query budget linters, enforce deterministic cryptographic schemas, cap serverless concurrency pools, and maintain strict hardware requirements before deploying customer-facing systems.

Related Tools & Operating Playbooks
FinOps Cloud Calculator →Killing Zombie Cloud Spend Playbook → All 10 Museum Exhibits →

Book a 1-on-1 Systems Architecture Review

INR 2,000 / $20 USD

Schedule a 45-minute private consultation with Arkajit Das. Review your technical architecture, prevent costly infrastructure mistakes, and optimize your systems.

Book Calendar Slot → PayPal $20 USD → Back to Failure Museum →