“Cost is not something you notice. It is something you go and look at.”
The bill was eight times larger than it needed to be
Three causes, of three different kinds, and not one of them produced an error.
The first thing we did on that project was read the cloud bill. It came to $24,000 a month. Six months later the same system ran for $2,000.
Nothing was decommissioned. The product served the same users doing the same work. The difference was three separate problems — one of sizing, one of breakage, one of architecture — and that distinction matters more than the number, because each kind is found a different way.
The problem
The constraint
What we did
We found the sizing problem by comparing provisioned capacity against observed use
Everything was over-provisioned — instances, database, the lot. This is rarely negligence: someone sizes an environment at launch when nobody knows the load, picks generously, and correctly so. Then the load becomes known and nobody returns, because nothing is wrong and something else is always more urgent.
Signal: low utilizationWe found the fault by looking for cost that didn’t track traffic
There were infinite loops running in the system — not a hot path, not an inefficient query, actual loops consuming compute continuously that somebody had been paying for month after month without knowing they existed. Over-provisioning shows up as low utilization. A runaway process shows up as high utilization nobody can attribute to anything a user is doing.
Signal: a flat curve overnightWe found the architectural cause by reading the code
Uploads were routed through the backend instead of going straight from the client to object storage. Every photograph — of a face, of a document — arrived at the server, occupied its memory for the duration of the request, and was then pushed onward. In a system where every application is an upload, that isn’t an inefficiency at the edge. It’s the main path.
Signal: none — only the code
Compare provisioned capacity against observed utilization over a month. If your database sustains 8% CPU and never peaks above 25%, that is not headroom, that is a decision nobody revisited. It persists because an over-provisioned system behaves perfectly: no symptom to notice, no alert to fire, no user to complain. The only signal is the invoice, and nobody reads the invoice.
The result
This bill was not 29% too large.
What we’d do differently
The bill should not have been the developer’s discovery. Nobody was watching it and nothing alerted when it grew. A budget alarm costs nothing and would have caught the infinite loops in their first week rather than their fiftieth.
- AWS
- S3
- Node.js