We got 50% faster — and why you should doubt that

The last four months of a four-year technical leadership engagement went into putting AI into the working process of a team of four developers. By the end the team was moving roughly 50% faster. That is the sentence that opens doors in a sales conversation, and it is the one we want to argue with.

Not because it is wrong. Because of how it was produced, and because of what people assume it means.

Where the number comes from

Jira tickets. Throughput before, throughput after, over comparable periods, on the same product, with the same four people. That is the whole methodology.

It is a reasonable operational measure and a weak scientific one. It does not isolate the model from everything else that changed in those four months, and a great deal changed in those four months. If you ask us how it was measured, that is the answer, and we would rather give it than round it into something firmer.

The trial that should make you cautious

In July 2025 METR published the most careful experiment on this question so far. Sixteen experienced open-source developers, 246 real issues from repositories they had maintained for years, each issue randomly assigned to allow or forbid AI tools.

Before starting, the developers expected AI to make them 24% faster. Afterwards, they estimated it had made them 20% faster. Measured, they took 19% longer.

The slowdown is not the interesting part. The gap is: 39 points between what the developers felt and what the clock recorded, among people with real experience of both the tools and the codebase.

It cuts both ways, and the honest reading has to include that too. METR revisited the design in February 2026 and found selection effects severe enough that they are rebuilding the experiment — among newly recruited developers the measured slowdown nearly disappears, and the organisation now marks the 2025 result as historical. So it is not evidence that AI makes teams slower. It is evidence that self-reported speed is unreliable, which includes ours.

What actually changed

If the 50% is real, we do not think the model earned most of it. The model arrived at the same time as a set of constraints, and the constraints are the part we would defend.

Engineering standards that had lived in review comments were rewritten as rules the build can check:

// the standard, as the build sees it
'@typescript-eslint/no-explicit-any': 'error',
'max-lines': ['error', { max: 200, skipBlankLines: true }],
'local/api-call-must-declare-request-and-response-types': 'error',

No any, and a prescribed way of declaring types. No component over 200 lines. Every API call carries a request type and a response type. A handful of others in the same spirit.

They run in two places:

# .husky/pre-commit — fast, staged files only
npx lint-staged

# ci — the same rules, no way around them
npm run lint -- --max-warnings=0
npm run typecheck
npm test

The hook is a courtesy: it answers the developer before they push. CI is the guarantee: nobody routes around it, including us. One is comfort, the other is the contract.

Why constraints matter more once a model is writing

A model produces plausible code, and plausible is the specific hazard. It reads well, it compiles, and it survives a review by someone who has already read four pull requests that morning. Human standards drift with fatigue, deadline and seniority. A rule that fails the build does not drift.

Before the rules, the standard was whatever the reviewer had the energy for that day. After, it was the same at nine in the morning and at seven in the evening, for the newest developer and for us. That is the change we would point to — and it happens to be the change that makes generated code safe to accept at speed.

Tests stopped being handwritten

Tests are the first thing dropped under deadline pressure. Once they stopped being manual work, they stopped being optional, and coverage stopped being a negotiation at the end of a sprint.

With one qualification worth stating: a generated test still needs a person to ask whether it tests the right thing. Left alone, a model will write a test that asserts the implementation rather than the behaviour, and it will pass forever without ever having been useful.

Where the model does the work

Components, tests, integrations, and code review against the written rules. Design goes through Claude Design connected to Figma, so a component and its mockup come from the same source rather than converging by hand. The team uses it every day, across the whole flow, not as an occasional lookup.

Business decisions are on the list too, and that one we would qualify: it drafts options and tradeoffs, and the decision stays with us.

What we would tell someone starting

Write the rules before you add the model. A model amplifies the process it lands in. If your standard lives in someone's head, you will get more code that varies with who reviewed it.

Enforce in two places. A hook for the answer within seconds, CI for the guarantee. Either one alone leaves a hole.

Pick a measure before you start, and know what it is worth. Ours is ticket throughput. It is honest about what it is, which is more than a number derived from asking the team how it felt.

Do not trust the feeling. That is the one thing the METR trial established past argument, and it applies to the people most convinced they are the exception.

What we will not claim

That the 50% transfers. It came from four developers who had worked together for years, in a codebase we had rebuilt ourselves, under rules written for that codebase. Remove any one of those and the number belongs to someone else's project, not yours.

What does transfer is the order of operations: standards first, made executable, then the model on top of them. That part costs a week and survives whichever tool you are using next year.

This came out of four years of technical leadership for a US marketplace, through four re-platformings and a team of twelve. Read the case study →