Cutting an AWS bill from $24,000 to $2,000 a month

Nothing in the system was broken in a way anyone would notice.

Cloud waste is hard to catch, and it persists, because it breaks nothing. A slow endpoint draws complaints and a failing job raises an alert, but an oversized instance, a runaway loop or a badly placed upload path keeps working normally, month after month, while you pay for it.

Until someone sets up a budget alarm, the only way to find this kind of waste is to go and look at the bill on purpose.

Over-provisioning. I found the sizing problem by comparing provisioned capacity with observed use. Everything was over-provisioned, including the instances and the database. This is rarely negligence: someone sizes an environment at launch when nobody knows the load, picks generously, and correctly so. Then the load becomes known and nobody goes back to it, because nothing is wrong and something else is always more urgent.

Infinite loops. I found the fault by looking for cost that didn't track traffic. Loops in the system were consuming compute continuously, and somebody had been paying for them month after month without knowing they existed. These were actual infinite loops, not a hot path or an inefficient query. Over-provisioning shows up as low utilization, while a runaway process shows up as high utilization that nobody can attribute to anything a user is doing.

Uploads routed through the backend. I found the architectural cause by reading the code. Uploads went through the backend on their way to S3, when the client could have sent them there directly. Every photograph, whether of a face or of a document, arrived at the server, occupied its memory for the duration of the request and was then pushed onward. In this system every application is an upload, so the upload path was the main path through the product.

Routing uploads through the backend tied instance memory to the number of concurrent uploads, sent every byte across the network twice, and capped throughput at what the application server could handle, although S3 is designed to absorb far more.

No dashboard shows a cause like this one, which is why it tends to be found last, and it is usually the largest of the three.

• Monthly spend: $24,000 → $2,000
• Against what the workload actually needed, the bill was about eight times too large, and the final bill came in lower still, so the cut from $24,000 to $2,000 is twelvefold
• Each of the three causes was found by a different method, and none of them raised an error

The bill should not have been the developer's discovery. Nobody was watching it and nothing alerted when it grew. A budget alarm costs nothing and would have caught the infinite loops in their first week rather than their fiftieth.