Three reasons your cloud bill is bigger than it should be

The first thing we did on that project was read the bill. Not the backlog, not the repository — the invoice. It came to about $24,000 a month for a system that, once we understood the traffic, had no business costing more than a few thousand. Six months later it was around $2,000, and no feature had been removed to get there.

The system issues passports and national ID cards for a national identity body. Agents go out to applicants who have no internet connection, so the application, the photographs and the biometrics are captured on a company phone and sent when a connection appears. That detail matters for what follows: in this system, almost every request is an upload.

Nobody asked us to reduce the bill. It was not in the scope. But an invoice is the one document in a project that no one can argue with, and reading it first told us more about the state of the system than a week of meetings would have.

Read the bill before you read the backlog

A bill is a measurement of what the system actually does, taken by a party with no stake in the story. Code review tells you what the code says. Monitoring tells you what the team decided to watch. The invoice tells you where the machine's time and bandwidth genuinely went, including the parts nobody remembers building.

It is also the fastest way to sort problems by type. Group the bill by service and by environment, then ask one question per line: is this the price of the work, or the price of a mistake? On this project three lines answered "mistake", and each was a different kind.

Cause one: everything was one size too big

Instances, database, workers — all provisioned generously, on an estimate made before the system had users. This is the most common cause and the least interesting one, because it is nobody's fault. You size infrastructure at a point when you have no data, and the honest choice is to over-provision. What goes wrong is that the estimate is never revisited once real numbers exist.

Over-provisioning has a signature: utilization that is not merely low but flat. A machine doing real work has a shape to its day. A machine that is too large has a straight line at 8% with a bump when the daily job runs.

Two rules we follow before resizing anything. Look at weeks, not hours — a single busy afternoon is not a capacity plan. And separate the peaks by cause: on this system the fleet had been sized for upload concurrency, not for request throughput, which means the machine was large because of buffers held in memory, not because of computation. That distinction decides whether you scale down or change the architecture, and here the answer turned out to be both.

Cause two: loops nobody noticed

This one is not optimization. It is a fault that had been running in production, consuming compute continuously, and had been on the invoice every month without anyone connecting the amount to a defect.

The shape was familiar: work that reschedules itself and has no terminal state. Something like this, in outline —

// a retry with no end condition and no dead letter
async function processPending() {
  const jobs = await db.jobs.findPending();

  for (const job of jobs) {
    try {
      await handle(job);
      await db.jobs.markDone(job.id);
    } catch (err) {
      // back to pending, forever, with no attempt count
      await db.jobs.markPending(job.id);
    }
  }

  setTimeout(processPending, 1000);
}

A job that can never succeed — a malformed record, a document that no longer exists, a third party returning a permanent error — is retried a hundred thousand times a day. It costs compute, it costs database connections, and it costs log ingestion, which on a large enough loop is the line that hurts most. Nothing is broken from the outside. The API responds. Nobody files a ticket. The invoice quietly grows.

The signature on a bill is the opposite of the first cause: cost that does not follow usage. If the graph is as high at four in the morning as at midday, something is running that is not driven by users. That is the question worth asking about any flat line on an invoice — not "can we make this cheaper" but "what is this, exactly".

The fix is not a smaller instance. It is an attempt counter, a terminal state, and somewhere for the failures to go.

const MAX_ATTEMPTS = 5;

async function handleJob(job) {
  try {
    await handle(job);
    return db.jobs.markDone(job.id);
  } catch (err) {
    const attempts = job.attempts + 1;

    if (attempts >= MAX_ATTEMPTS || isPermanent(err)) {
      // stops costing money and starts being visible
      return db.jobs.markFailed(job.id, err.message);
    }

    return db.jobs.retryIn(job.id, attempts, backoff(attempts));
  }
}

The important line is markFailed. A queue with no failed state does not eliminate failure, it converts it into recurring cost. Exponential backoff caps the rate; the attempt limit caps the total; a failed row is something an operator can see, which is the point. We would rather have fifty rows in a failed state and one alarm than a loop that is invisible everywhere except on the invoice.

Cause three: every byte went through the backend

Every uploaded file — a photograph of a face, a photograph of a document — travelled from the phone into the Node process and from there to object storage. In a system where nearly every request carries an image, this was not a rough edge on the architecture. It was the architecture, and it was the largest single line on the bill.

Routing files through your own server costs four things at once: memory for the buffers, bandwidth for bytes that travel twice, an instance size chosen for upload concurrency rather than for work, and availability — because uploads and API calls share the same workers, so a burst of slow uploads on a bad mobile connection makes fast endpoints queue behind them.

Moving to pre-signed uploads takes the bytes out of the request path entirely. The change itself is small; the design work is deciding where validation and authorization live once your server never sees the file. We wrote that up separately, with the code and the traps: direct S3 uploads, and where the checks go afterwards.

Why the order mattered

All of this happened before we built a single feature. That was deliberate, and it is the same judgement we apply to inherited systems generally: stabilize what is there before adding to it. A system that is quietly wasting eight times its cost is telling you something about how it was operated, and that context is worth having before you commit to a plan.

There is a practical argument too. Fixing the bill bought trust in the first weeks, with a number nobody had to take on faith. Every architectural conversation after that started from a different place.

And the sequence protected the work: had we built features first, the upload path would have been baked into three more flows, and cause three would have gone from a two-day change to a migration.

What we would do differently

Put a number on each cause, not on the total. We know the bill went from about $24,000 to about $2,000. We cannot tell you how much of that was the resizing, how much the loops, and how much the uploads. Three separate measurements would have cost us an hour each and would have made this article better — and, more usefully, would have told the client which class of mistake their system was most prone to.

Alarm on cost per environment, from day one. Not a monthly budget alert on the account, which arrives after the money is spent, but a daily anomaly alarm per environment. The loops were discoverable on any day of the six months they ran; nothing was watching.

Tag resources so the bill can answer questions. Grouping by service tells you that compute is expensive. Grouping by environment and by feature tells you which part of the product is expensive, which is the question anyone actually has.

If there is one thing to take from this: separate the three kinds. Something mis-estimated is a decision to revisit. Something broken is a defect with a cost attached. Something built the wrong way round is an architectural change with a deadline attached, because it gets more expensive to fix with every feature that leans on it.

This came out of building a national ID and passport issuance system, where the same review also found uploads routed through the backend and loops nobody noticed. Read the case study →