I served ResNet-50 on a CPU and stopped paying for GPUs

When I started the attendance system I assumed I would need a GPU. Face recognition, convolutional networks, real-time video — every instinct said accelerator.

The final build runs a ResNet-50-based recognition model and a second anti-spoof CNN on a laptop with no discrete GPU, and hits its latency budget with room to spare. The assumption was wrong, and it was wrong in a way I think is common.

Start from the budget, not the hardware

The question "do I need a GPU?" is unanswerable. The answerable question is "what is my latency budget, and what fits inside it?"

For this system the pipeline processed roughly one frame every 350 milliseconds. That is the real constraint, and it is generous — it comes from how fast a human moves their head, not from anything technical.

Once the budget is explicit, the work becomes measurement rather than argument. What does each stage cost? Does the total fit? If it fits on the hardware you already have, the GPU question is closed.

Where the time actually went

Per frame, measured rather than estimated: JPEG decode 3.1ms, facial landmark inference 9.8ms, anti-spoof CNN 11.4ms, screen-artefact FFT analysis 5.3ms, depth-motion update 1.8ms, camera-authenticity update 1.1ms. Total: about 32.5ms, against a 350ms budget. Transport — browser capture, HTTP, JSON serialisation — added another 14 to 18ms.

Two things stand out.

There was an order of magnitude of headroom. Not 10% spare, roughly 10× spare. Any conversation about acceleration was over before it began.

And the expensive operation was not per frame. The recognition forward pass costs about 92.5ms — three times any single per-frame stage — but it runs once per session, not thirty times a second. Recognising that, and structuring the pipeline so the costly model runs once while cheap detectors run continuously, is what made the whole thing viable. It was an architecture decision, not an optimisation.

Most "we need a GPU" conclusions are really "we did not check which part was slow, or how often it actually runs."

What ONNX Runtime buys you

Exporting to ONNX and serving through ONNX Runtime did three useful things.

It produced a portable graph, so the training stack and the serving stack stopped being the same decision. The service does not carry PyTorch at all, which removed a large dependency and a large slice of container image.

It applied graph-level optimisations — operator fusion, constant folding, dead node elimination — that a naive framework forward pass does not.

And it used CPU-tuned kernels. This is the part people underestimate. Modern CPU inference is not the naive loop people picture; it is heavily vectorised, cache-aware code, and the gap to a GPU on a single small input is far smaller than the gap on a large batch.

That last point is the crux. GPUs win on throughput through parallelism. If you are processing one image at a time in response to a user action, you are not using the thing a GPU is good at. You are paying for parallel hardware to do serial work.

Quantisation, and when to stop

Quantisation — moving weights from 32-bit floats to 8-bit integers — typically buys meaningful speed and a big reduction in model size, for some accuracy cost.

I did not need it here, and that is the point worth making. With 10× headroom, quantising would have traded accuracy for latency I had no use for. Optimisation you do not need is not free; it costs accuracy, complexity, and a validation cycle to prove you did not break anything.

Measure first. If you fit, stop.

The economics

A modest GPU instance runs several times the cost of a comparable CPU instance, and that difference compounds every hour of every month whether or not you are serving traffic.

The CPU deployment also removed operational surface: no driver versions, no CUDA compatibility matrix, no accelerator-specific base images, no scheduling constraints. The service runs anywhere a container runs. For a system meant to be deployed by a university IT department rather than a platform team, that mattered more than the money.

When you genuinely do need the GPU

Training, essentially always. Large-batch throughput where parallelism is the actual workload. Models where a single forward pass exceeds your entire latency budget on CPU — large language models being the obvious case, where the arithmetic is not close.

Those are real and common. They are just far less universal than the default assumption implies, and the cost of checking is one afternoon with a profiler. I have now done that check several times and been surprised in the same direction more often than not.