Taking over a stalled model: 49% to 77.7% recall

A professor asked me to look at a skin-lesion classification project that had stopped progressing. The pipeline ran, the model trained, the accuracy figure was respectable, and it was close to useless.

Accuracy was the wrong objective

The dataset was over 27,000 dermoscopy images, heavily imbalanced toward benign lesions — which is a faithful reflection of reality and a trap for anyone optimising accuracy.

On imbalanced data, a model that leans toward the majority class posts a strong accuracy number almost for free. Predict "benign" often enough and you are right most of the time, because most lesions are benign. Meanwhile recall on the class that matters — correctly identifying malignant lesions — sat at 49%. The model was missing more than half of what it existed to find.

In screening, the two errors are not symmetric and it is not close. A false positive costs a second look by a clinician: some time, some anxiety, a follow-up. A false negative is a missed cancer. Any metric that treats those as equivalent is not measuring the thing the system is for.

The four changes that moved it

None of them exotic.

**Focal loss.** Standard cross-entropy lets a large number of easy, confidently correct majority-class examples dominate the gradient. Focal loss down-weights examples the model already gets right, so the learning signal comes from the hard and rare cases.

**Class-weighted training**, so a malignant example contributes proportionally more to the loss than a benign one — encoding the asymmetry of the errors into the objective rather than hoping the model infers it.

**Stratified augmentation.** The original splits did not preserve class distribution, which meant validation numbers were partly measuring how the split fell. Stratifying made validation trustworthy. This changed no model behaviour at all and changed every decision made afterwards, because we were finally reading real numbers.

**Architecture search** with a mobile-scale backbone, tuned with a Keras Tuner sweep rather than a hand-picked configuration. This contributed the least of the four, which I did not expect — the loss and sampling work mattered far more than the architecture.

Rebuilding a stalled project is usually less about better modelling and more about noticing that everyone has been optimising the wrong number.

Where it landed

Recall on malignant lesions went from 49% to 77.7%, at 0.86 AUC.

Overall accuracy went down slightly. That is the correct trade for this problem, and it needs saying out loud whenever you report it — otherwise the next person reads the accuracy column, concludes the model got worse, and reverts your work.

This is why I now report the metric the system exists to optimise first and prominently, with the others as supporting context. Leading with accuracy on an imbalanced medical problem is close to misinformation, even when every number is true.

The threshold is a product decision, not a default

One more thing mattered and is easy to miss: the classification threshold.

Models output a probability. Turning that into a decision requires a cutoff, and the default of 0.5 is a convention, not a recommendation. It happens to be right only when the two errors cost the same — which, in screening, they emphatically do not.

Sweeping the threshold and plotting recall against precision makes the trade explicit. You can then choose the operating point deliberately: for a screening aid that flags cases for a clinician to review, high recall matters and a moderate false-positive rate is tolerable, because the cost of a false positive is a second opinion.

The same trained model can be a useful screening tool or a useless one depending purely on where that line is drawn. Reporting a single accuracy figure hides the fact that a choice was made at all — usually by accident, by whoever left the default in place.

What I would do differently

I should have plotted the per-class confusion matrix on day one instead of week two. It would have shown the problem immediately and unambiguously. Instead I spent the first stretch investigating training dynamics and architecture — reasonable-sounding work that was never going to help, because the constraint was elsewhere.

There is a general habit in that. When inheriting a stalled model, before touching anything: look at per-class performance, check whether validation is stratified, and ask what metric the system is actually being judged on in the real world. All three are quick. Any one of them can save a fortnight.

On inheriting other people's work

The original pipeline was not bad. It was competent code aimed at a target that did not match the clinical problem — an easy mistake, because "maximise accuracy" is the default framing in almost every tutorial and benchmark.

The most valuable contribution was not modelling skill. It was asking what a good outcome looked like to a clinician, and then noticing that the loss function had never been told.