Skin-tone fairness auditing for dermatology AI
Dermatology AI is not
validated on dark skin.
We test whether a skin-cancer AI works as well on dark skin as it does on light skin. A vendor sends a table of their model’s predictions — no images, no weights, no patients. We send back what it can do per skin tone, where the data cannot support an answer at all, and what it would cost to fix that.
PAD-UFES-20 · 2,298 images · 1,373 patients · CC BY 4.0
Here is why nobody can answer that question today. Below is every image in the best openly-licensed dermatology dataset that has both biopsy-confirmed cancer labels and skin tone — one dot each.
Find the dark ones.
There are 11. Out of 2,298. They carry 8 cancers between them.
A further 804 images (35%) carry no skin-tone label at all, and can never be audited for tone by anyone, with any tool.
What this is
We test whether a skin-cancer AI works as well on dark skin as it does on light skin.
Regulators and health-service procurement have started asking dermatology-AI vendors to prove their models perform across skin tones. The vendors cannot — most have no way to even measure it, and none publishes the answer. We are the instrument that measures it.
- Who pays
- AI vendors, health services buying an independent audit, and university groups who need per-subgroup performance for a paper.
- What they hand over
- Nothing. No images, no model weights, no patient records — just a table of numbers. That is what makes this safe for a competitor to buy, and safe for a hospital to permit.
- 1
They send a table
A CSV of their model's own predictions: was it cancer, what did the model say, what was the patient's skin tone. That is all.
id, y_true, y_score, tone - 2
We audit it
Performance per skin tone, each figure with a confidence interval — and an explicit refusal wherever the data cannot support a number.
- 3
They get the evidence
A dated, procurement-ready pack: what was measured, what could not be, and exactly how many more cases it would take to close the gap.
And what this is not
- Not a diagnosis. It never touches a patient. It audits models, not people — which is also what keeps it outside medical-device regulation.
- Not a competitor to MoleMap or any other vendor. It is the ruler you hold against them, and it is one they can buy.
- Not a claim to be saving anybody yet. The audit does not treat a single patient. It measures a gap that has never been measured, and prices what it would take to close it. That is a smaller claim than the one this field usually makes, and it is one we can actually stand behind.
What our own tool says about our own model
We trained a standard skin-cancer classifier on that data and pointed our auditor at it. It refused to certify a single skin tone — including the ones that flatter us.
Every other tool would print four confident numbers here. Ours prints one and hedges it — because the dataset holds 12 benign controls on light skin, nine on mid, and 1 cancer case on dark. That is what an audit is for.
0.864
AUROC · 95% CI 0.681–0.980
- Images
- 170
- Suspicious
- 158
- Calib. err
- 0.076
declined
71 images · 62 suspicious
- Images
- 71
- Suspicious
- 62
- Calib. err
- —
declined
2 images · 1 suspicious
- Images
- 2
- Suspicious
- 1
- Calib. err
- —
declined
118 images · 69 suspicious
- Images
- 118
- Suspicious
- 69
- Calib. err
- —
| Skin tone | Images | Suspicious | AUROC | 95% CI | Sens @ 90% spec | Calib. error | Evidence |
|---|---|---|---|---|---|---|---|
| Fitzpatrick I–II (light) | 170 | 158 | 0.864 | 0.681–0.980 | undefined | 0.076 | Indicative · n≥10 |
| Fitzpatrick III–IV (mid) | 71 | 62 | declined | — | undefined | — | Not measured |
| Fitzpatrick V–VI (dark) | 2 | 1 | declined | — | undefined | — | Not measured |
| No skin-tone label | 118 | 69 | declined | — | undefined | — | No tone label |
Fitzpatrick V–VI (dark) — 1 case. Too few to state any number, so none is stated. That is a fact about the data, not about the model — and it is true of every model tested on it.
Why each bin says what it says
Fitzpatrick I–II (light) — Sensitivity at 90% specificity is UNDEFINED for this bin, not zero. With 12 benign case(s) the specificity axis moves in steps of 8%, and no operating point that flags any lesion at all reaches 90%. The bin needs more BENIGN controls before this question can be asked of it — not more cancers. 158 positive and 12 benign case(s): enough to point, not enough to certify (the bar is 30 of EACH). Read the confidence interval, not the point estimate. The AUROC interval spans 0.30 — too wide to distinguish a good model from a poor one in this bin.
Fitzpatrick III–IV (mid) — Only 62 positive case(s) in this bin — below the 10 needed even for a directional read. No AUROC, sensitivity, or gap is reported for this bin, because any number computed on 62 case(s) would carry a confidence interval wide enough to span the whole plausible range. THIS IS THE FINDING: the model is unaudited on this skin tone. It is not a claim that the model is bad here, and it must not be read as one — it is a statement that nobody knows.
Fitzpatrick V–VI (dark) — Only 1 positive case(s) in this bin — below the 10 needed even for a directional read. No AUROC, sensitivity, or gap is reported for this bin, because any number computed on 1 case(s) would carry a confidence interval wide enough to span the whole plausible range. THIS IS THE FINDING: the model is unaudited on this skin tone. It is not a claim that the model is bad here, and it must not be read as one — it is a statement that nobody knows.
No skin-tone label — 118 prediction(s) carry no skin-tone label. They cannot be audited for tone by this or any other tool. Tone-label coverage is itself an audit finding: a model whose evaluation set is 33% tone-unlabelled has not been shown to work across tones, whatever its headline metric says.
“Fine. How many cases would it take?”
A refusal that stops there is useless. So the audit prices it — a line item a health board can budget, not an aspiration.
239 more cancer case(s) and 386 more benign case(s) stand between this evaluation set and a model that can be certified to ±0.05 on every skin tone. Of that, 150 cancer(s) on Fitzpatrick V-VI skin — the data that does not exist anywhere, at any price, under a licence that permits it to be used commercially.
| Skin tone | Has | Supports | Needs | Shortfall | Bottleneck |
|---|---|---|---|---|---|
| Fitzpatrick I–II (light) | 158 / 12 | ±0.077 | 106 / 106 | +0 / +94 | needs benign controls |
| Fitzpatrick III–IV (mid) | 62 / 9 | ±0.127 | 151 / 151 | +89 / +142 | needs both |
| Fitzpatrick V–VI (dark) | 1 / 1 | ±0.784 | 151 / 151 | +150 / +150 | needs both |
| No skin-tone label | 69 / 49 | ±0.078 | — | — | needs tone labels |
Counts are positives / negatives. “Supports” is the 95% interval the current sample can actually sustain; “Needs” is what a ±0.05 interval would require at 1:1.
Fitzpatrick V–VI (dark) — With 1 positive and 1 benign case(s) this bin supports a 95% interval of only ±0.78 — wider than the plausible range of the metric itself. To estimate auroc to within ±0.05 at a assumed value of 0.80, it needs roughly 151 positive and 151 benign case(s) at 1:1, against 1 and 1 today. SHORTFALL: 150 cancer(s) and 150 benign case(s). The projection assumes auroc ≈ 0.80, because this bin has never been measured — that is the assumption to argue with, and the requirement moves if you change it.
The requirement for the other tones
Fitzpatrick I–II (light) — With 158 positive and 12 benign case(s) this bin supports a 95% interval of only ±0.08 — too wide to act on. To estimate auroc to within ±0.05 at a measured value of 0.86, it needs roughly 106 positive and 106 benign case(s) at 1:1. It has enough cancers (158) — what it lacks is BENIGN controls. SHORTFALL: 94 benign case(s). This is the cheap, low-risk half of the ask: the bottleneck here is healthy skin, not more cancer.
Fitzpatrick III–IV (mid) — With 62 positive and 9 benign case(s) this bin supports a 95% interval of only ±0.13 — too wide to act on. To estimate auroc to within ±0.05 at a assumed value of 0.80, it needs roughly 151 positive and 151 benign case(s) at 1:1, against 62 and 9 today. SHORTFALL: 89 cancer(s) and 142 benign case(s). The projection assumes auroc ≈ 0.80, because this bin has never been measured — that is the assumption to argue with, and the requirement moves if you change it.
No skin-tone label — These 118 prediction(s) carry no skin-tone label, so no amount of additional data fixes them — the images themselves would have to be re-labelled for tone. Labelling is the intervention here, not collection.
How these numbers are derived, and what they do not promise
AUROC precision from the Hanley & McNeil (1982) variance estimator. These are planning estimates under a normal approximation, not guarantees, and the requirement depends on the true metric value — which is unknown before you measure it, so unmeasured bins are projected at an explicitly-stated assumed value. A larger sample buys precision, not representativeness: it cannot repair a biased cohort, and it cannot repair a consent regime that forbids the intended use.
So we broke the rule, in a lane that can never ship
There is a dataset with dark skin in it. It is licensed non-commercially, so a model built on it can be published and never sold. We trained one anyway, to find out what the industry has never been able to measure.
Lane 1 · we may sell this
Trained on the only data we are allowed to ship.
PAD-UFES-20 · CC BY 4.0 · commercially licensed. It contains 11 dark-skin images in total — 2 of them land in the held-out test split.
- Light skin (I–II)
- 0.86495% CI 0.681–0.980
- Dark skin (V–VI)
- declined1 cancer case in the test set
The auditor refuses to score it on dark skin. Not because the model failed — because there is nothing to test it with. Every competitor shipping today is in exactly this position, and none of them says so.
Lane 2 · we may never sell this
Trained on data that actually contains dark skin.
Fitzpatrick17k · CC BY-NC-SA · non-commercial. It contains 93 dark-skin cancers — enough to certify.
- Light skin (I–II)
- 0.96995% CI 0.931–0.994
- Dark skin (V–VI)
- 0.98295% CI 0.944–1.000
This evaluation set does not establish an accuracy gap. The difference is -0.013 (95% CI -0.059 to 0.033), and the interval straddles zero — so it does not rule a gap out either. What it does show is that a model given dark-skin cases performs comparably on them. Absence of evidence is not evidence of absence, and we will not claim the stronger thing.
The evidence points at the data, not the algorithms — and the data that would settle it cannot legally be sold.
Give a model dark-skin cases and it performs comparably on dark skin. So the constraint is data — and the data that would settle it is data nobody is allowed to sell.
Two things we found that we did not want to find
The model ranks dark lesions correctly. Its confidence still lies.
Accuracy was equal across tones — but calibration error was worst on dark skin. A model can order lesions perfectly and still be badly over-confident about them, and a triage threshold is applied to the probability, not to the ranking. So an over-confident model sends dark-skinned patients home. An audit that only checked AUROC would have called this model fair and been wrong.
The dashed diagonal is honesty: of everything the model called 70% likely, 70% should turn out to be cancer. Points in the shaded half claim more confidence than they earn. AUROC is mathematically invariant to all of this — it cannot see the failure at all. The per-tone calibration error is in the legend; dark skin carries the worst of it, on the fewest cases.
And we could not fix it. Fitting the correction for dark skin needs dark-skin cases, and there were 14. The tool refused to fit rather than apply a correction it could not validate — after an earlier version of it, fitted on that same handful of cases, made dark-skin calibration measurably worse.
The field’s own fairness benchmark is disappearing.
Fitzpatrick17k — the dataset the dermatology-AI fairness literature cites to demonstrate skin-tone bias — ships URLs, not images. When we fetched it, 75.9% of its cancer images were gone: one of its two host sites now returns 404 for every single file. Of 4,497 images we could retrieve 1,083.
The benchmark used to prove that dermatology AI is biased is no longer reproducible as published. The commercial data never had dark skin; the research data that did is rotting away. Both roads lead to the same place.
A third failure, and no fairness metric catches it
The same model is right 19 times out of 20 in a clinic — and wrong 19 times out of 20 in screening.
Not two models. Not two thresholds. The same weights, at the same operating point (sensitivity 0.93, specificity 0.83), pointed at two different populations.
Positive predictive value is not a property of a model. It is a property of a model and the people it is used on, and it collapses as cancer gets rarer. Every one of those false alarms in a screening setting is a frightened patient, and often a scar.
No fairness metric catches this. AUROC, sensitivity and specificity are all prevalence-invariant — which is precisely why this audit leads with them, and never prints a PPV without the population it was computed in. A vendor who benchmarks on a clinic cohort and quotes that PPV in a screening pitch has not lied about a number. They have quoted a real number from the wrong world.
| Where the model is used | PPV | NPV |
|---|---|---|
| GP / primary care screening | 0.05 | 1.00 |
| teledermatology triage | 0.23 | 1.00 |
| specialist referral clinic | 0.58 | 0.98 |
| skin-cancer treatment cohort | 0.95 | 0.76 |
Computed for I-II at its operating point, by Bayes from sensitivity and specificity. Both of those are measured; the prevalences are the settings a dermatology model actually gets deployed into.
And the model is over-confident, which AUROC also cannot see
Calibration slope 0.23 (1.00 is honest). Below 1 means the predictions are too extreme — the model calls the safe lesions safer than they are and the dangerous ones worse. It is the classic over-fitting signature, it is what per-tone temperature scaling repairs, and a report that only quoted AUROC would never have mentioned it.
Our own reference model, audited by our own tool. We publish this because it is the kind of thing an audit is for— and because a company that only ran this analysis on other people’s models would deserve the scepticism it got.
The lesion that matters most here is not in the data at all
Acral melanoma — on the soles, the palms, under the nails — is the presentation that disproportionately affects Māori and Pacific patients.
It grows on skin that never sees the sun. It is not what a clinician is trained to look for. It is caught late. In a country where Māori present with thicker tumours at later stage, it is the single most equity-relevant lesion type there is — and so it is the one subgroup this project is most obliged to be able to audit.
In the entire dataset there are 142 lesions on an acral site, and 2 of them are melanoma. On dark skin, there are none.
Not few. Not under-represented. Zero. The lesion that most often kills the patients this venture exists to serve does not appear, on their skin, in the best openly-licensed dermatology data in the world. No model can be validated for it — not ours, and not anyone’s.
| Acral site | Images | Suspicious | Melanoma |
|---|---|---|---|
| Fitzpatrick I–II | 60 | 56 | 2 |
| Fitzpatrick III–IV | 28 | 25 | 0 |
| Fitzpatrick V–VI | 0 | 0 | 0 |
| No tone label | 54 | 47 | 0 |
And it is worse than the zero
The dataset records body sites as HAND and FOOT — never PALM, SOLE, or NAIL. The back of the hand is not an acral site; the palm is. So even the 142 images we do have cannot isolate a true acral lesion.
The vocabulary needed to ask the question does not exist in the data. This is not a gap a better model closes. It is a gap a dataset closes — consented, sovereignty-governed, and collected from the people who actually present with it.
Run your own model through it. Nothing leaves your browser.
Drop in a CSV of predictions — y_true, y_score, tone — and the audit runs locally, in your tab. We never receive the file. You get the same per-tone table, the same confidence intervals, and the same refusals you see above.