Skin-tone fairness audit

NZ Skin Equity reference model (efficientnet_b0) · trained 13 epochs · NZ Skin Equity (self-audit) — generated 2026-07-13

On Fitzpatrick I–II the model reaches AUROC 0.864 (95% CI 0.681–0.980). On Fitzpatrick V–VI it has 2 evaluation case(s), of which 1 are positive — too few to state any performance number at all. This model is NOT AUDITED on dark skin. That is not a finding about the model; it is a finding about the data that exists to test it with, and it applies to every model evaluated on this dataset, including every competitor's.

1 · What was audited

Model	NZ Skin Equity reference model (efficientnet_b0) trained 13 epochs
Task	binary lesion triage (benign vs suspicious)
Evaluation set	PAD-UFES-20 held-out test split (patient-level)
Dataset licence	CC BY 4.0
Report schema	v1.0

This is our own model, audited by our own engine, on a public dataset. It is published so the method can be checked rather than taken on trust — including the part where the tool declines to certify our own model on dark skin.

This is a confidence-interval-reported audit, not a pass/fail certificate. No fairness "certificate" is issued here, by design. A bright-line pass/fail would imply a precision the statistics do not support, and it is exactly the artifact a vendor could wave at a regulator to end a conversation that ought to continue. A bin is certified at ≥30 positive cases, called directional at ≥10, and left unreported below that.

2 · Per-tone performance (uncorrected model)

Skin tone	n	positives	AUROC	95% CI	Sens@90% spec	ECE	Certification
Fitzpatrick I–II (light)	170	158	0.864	0.681 – 0.980	not reported	0.076	DIRECTIONAL ONLY
Fitzpatrick III–IV (mid)	71	62	not reported	—	not reported	not reported	INSUFFICIENT DATA — NOT AUDITED
Fitzpatrick V–VI (dark)	2	1	not reported	—	not reported	not reported	INSUFFICIENT DATA — NOT AUDITED
No skin-tone label	118	69	not reported	—	not reported	not reported	NO TONE LABEL — NOT AUDITABLE

Fitzpatrick I–II (light) — Sensitivity at 90% specificity is UNDEFINED for this bin, not zero. With 12 benign case(s) the specificity axis moves in steps of 8%, and no operating point that flags any lesion at all reaches 90%. The bin needs more BENIGN controls before this question can be asked of it — not more cancers. 158 positive and 12 benign case(s): enough to point, not enough to certify (the bar is 30 of EACH). Read the confidence interval, not the point estimate. The AUROC interval spans 0.30 — too wide to distinguish a good model from a poor one in this bin.

Fitzpatrick III–IV (mid) — Only 62 positive case(s) in this bin — below the 10 needed even for a directional read. No AUROC, sensitivity, or gap is reported for this bin, because any number computed on 62 case(s) would carry a confidence interval wide enough to span the whole plausible range. THIS IS THE FINDING: the model is unaudited on this skin tone. It is not a claim that the model is bad here, and it must not be read as one — it is a statement that nobody knows.

No skin-tone label — 118 prediction(s) carry no skin-tone label. They cannot be audited for tone by this or any other tool. Tone-label coverage is itself an audit finding: a model whose evaluation set is 33% tone-unlabelled has not been shown to work across tones, whatever its headline metric says.

Fitzpatrick V–VI (dark) — Only 1 positive case(s) in this bin — below the 10 needed even for a directional read. No AUROC, sensitivity, or gap is reported for this bin, because any number computed on 1 case(s) would carry a confidence interval wide enough to span the whole plausible range. THIS IS THE FINDING: the model is unaudited on this skin tone. It is not a claim that the model is bad here, and it must not be read as one — it is a statement that nobody knows.

3 · The gap

I-II vs V-VI (auroc)
The V-VI bin holds 2 prediction(s) and 1 positive case(s) — against 170 and 158 in I-II. That is too few to estimate a auroc gap at any useful precision, so NO GAP IS REPORTED. The absence of a number here is the result: this model's behaviour on V-VI skin is unmeasured, and the data to measure it does not exist in the evaluation set provided.

I-II vs III-IV (auroc)
I-II minus III-IV auroc = +0.002 (95% CI -0.214 to +0.272). The interval crosses zero, so this evaluation set does NOT establish a gap — it also does not rule one out. Distinguishing them needs more III-IV cases, not a better model.

4 · After per-tone calibration

The same evaluation set, re-scored through the fitted calibration layer. The model's weights are unchanged and its ranking is unchanged — AUROC is mathematically invariant under temperature scaling, so any movement you see in AUROC would indicate a bug, not an improvement. What calibration moves is ECE: how honest the probabilities are.

Skin tone	n	positives	AUROC	95% CI	Sens@90% spec	ECE	Certification
Fitzpatrick I–II (light)	170	158	0.868	0.691 – 0.980	not reported	0.065	DIRECTIONAL ONLY
Fitzpatrick III–IV (mid)	71	62	not reported	—	not reported	not reported	INSUFFICIENT DATA — NOT AUDITED
Fitzpatrick V–VI (dark)	2	1	not reported	—	not reported	not reported	INSUFFICIENT DATA — NOT AUDITED
No skin-tone label	118	69	not reported	—	not reported	not reported	NO TONE LABEL — NOT AUDITABLE

5 · What it would take — the collection requirement

239 more cancer case(s) and 386 more benign case(s) stand between this evaluation set and a model that can be certified to ±0.05 on every skin tone. Of that, 150 cancer(s) on Fitzpatrick V-VI skin — the data that does not exist anywhere, at any price, under a licence that permits it to be used commercially.

Every refusal above is a missing measurement, and a missing measurement has a price. This table is that price, per skin tone, at a target 95% interval of ±0.05 on auroc. It is what a data partner would have to supply for this model to be auditable across tones — not an aspiration, a line item.

Read the bottleneck column before acting on this. AUROC precision depends on both classes, so a bin can be rich in cancers and still imprecise for want of benign controls — which is exactly the situation on light skin in a treatment cohort. Sending a partner to collect more melanoma when what the estimate lacks is healthy skin would be expensive, slow, and wrong.

Skin tone	Have (pos/neg)	Supports	Needs (pos/neg)	Shortfall	Bottleneck
Fitzpatrick I–II (light)	158 / 12	±0.077	106 / 106	+0 / +94	needs benign controls
With 158 positive and 12 benign case(s) this bin supports a 95% interval of only ±0.08 — too wide to act on. To estimate auroc to within ±0.05 at a measured value of 0.86, it needs roughly 106 positive and 106 benign case(s) at 1:1. It has enough cancers (158) — what it lacks is BENIGN controls. SHORTFALL: 94 benign case(s). This is the cheap, low-risk half of the ask: the bottleneck here is healthy skin, not more cancer.
Fitzpatrick III–IV (mid)	62 / 9	±0.127	151 / 151	+89 / +142	needs both
With 62 positive and 9 benign case(s) this bin supports a 95% interval of only ±0.13 — too wide to act on. To estimate auroc to within ±0.05 at a assumed value of 0.80, it needs roughly 151 positive and 151 benign case(s) at 1:1, against 62 and 9 today. SHORTFALL: 89 cancer(s) and 142 benign case(s). The projection assumes auroc ≈ 0.80, because this bin has never been measured — that is the assumption to argue with, and the requirement moves if you change it.
Fitzpatrick V–VI (dark)	1 / 1	±0.784	151 / 151	+150 / +150	needs both
With 1 positive and 1 benign case(s) this bin supports a 95% interval of only ±0.78 — wider than the plausible range of the metric itself. To estimate auroc to within ±0.05 at a assumed value of 0.80, it needs roughly 151 positive and 151 benign case(s) at 1:1, against 1 and 1 today. SHORTFALL: 150 cancer(s) and 150 benign case(s). The projection assumes auroc ≈ 0.80, because this bin has never been measured — that is the assumption to argue with, and the requirement moves if you change it.
No skin-tone label	69 / 49	±0.078	— / —	—	needs tone labels
These 118 prediction(s) carry no skin-tone label, so no amount of additional data fixes them — the images themselves would have to be re-labelled for tone. Labelling is the intervention here, not collection.

AUROC precision from the Hanley & McNeil (1982) variance estimator; sensitivity from the binomial. Both are planning estimates under a normal approximation, not guarantees, and the requirement depends on the true metric value — which is unknown before you measure it, so unmeasured bins are projected at an explicitly-stated assumed value. Note that sensitivity precision depends on POSITIVE cases only: collecting more benign images does not improve it. And a larger sample buys precision, not representativeness — it cannot repair a biased cohort or a consent regime that forbids the intended use.

6 · Calibration layer

Post-hoc per-tone temperature scaling, fitted on a held-out calibration split (calibration split). It changes no weights and requires no retraining. It corrects over-confidence; it does not create competence. Where a bin held too few positives, the fit is refused rather than made on noise.

Skin tone	positives used	Temperature	ECE before	ECE after	Δ
Fitzpatrick I–II (light)	145	3.270	0.057	0.060	-0.003
Fitted on 145 positive case(s) and validated out-of-sample (+0.1784 mean held-out log-loss gain over 3 folds).
Fitzpatrick III–IV (mid)	59	refused	0.030	—	—
FIT REJECTED: with 59 positive case(s), a fitted temperature did NOT improve the model out-of-sample (mean held-out log-loss change -0.0321 across 4 folds, below the +0.0020 required). Applying it would risk making this bin worse while appearing to fix it, so it is not applied. Predictions in this bin fall back to the global temperature (T=2.830). A correction that cannot demonstrate, on data it has not seen, that it makes calibration better is not applied — an unvalidated 'fix' that quietly degrades the group it was meant to help is the precise failure this tool exists to catch. AND THE GLOBAL TEMPERATURE IS NOT APPLIED EITHER. Checked against this bin it makes calibration WORSE (0.030 -> 0.073), so it is rejected and the bin runs UNCORRECTED (T=1.0). There is no correction we can honestly offer this group: we cannot fit one from their own data because there is too little of it, and the one fitted from everyone else's data harms them. That is the state of the art, stated plainly rather than papered over.
Fitzpatrick V–VI (dark)	2	refused	0.252	—	—
REFUSED TO FIT: 2 positive case(s), below the 30 needed. A temperature fitted on this few cases would track noise and could make calibration worse, while looking like a fix. Predictions in this bin fall back to the global temperature (T=2.830), fitted across all tones — which means this tone is running on a correction derived from other people's skin. That is the honest state of the art, and it is the gap the venture exists to close. The global temperature is NOT applied either: this bin has no usable mix of cases to check it against, so it runs UNCORRECTED (T=1.0).
No skin-tone label	66	2.850	0.061	0.064	-0.003
Fitted on 66 positive case(s) and validated out-of-sample (+0.1209 mean held-out log-loss gain over 5 folds).

7 · Limitations — read these before quoting any number above

118 of 361 evaluation cases (33%) carry no skin-tone label. They are excluded from every per-tone number in this report, and no tool — this one or any other — can audit them for tone. Tone-label coverage is itself an audit finding.
Fitzpatrick V–VI (dark): 1 positive case(s) — below the 10 required for even a directional estimate. No metric is reported for this bin. The correct reading is 'unmeasured', never 'fine'.
Fitzpatrick III–IV (mid): 62 positive case(s) — below the 10 required for even a directional estimate. No metric is reported for this bin. The correct reading is 'unmeasured', never 'fine'.
Calibration is fitted post-hoc and does not give the model new information about skin tones it has never seen. It corrects over-confidence; it does not create competence. Where a bin was too small to fit, predictions fall back to a temperature fitted across all tones — i.e. a correction derived from other people's skin.
All numbers are conditional on the evaluation set (PAD-UFES-20 held-out test split (patient-level)) being representative of the deployment population. Where it is not — different cameras, different lesion mix, a screening rather than a treatment cohort — these numbers will not transfer.
Stratified by Fitzpatrick (phototype — burn/tan response, NOT a colour scale). This is worth stating plainly: Fitzpatrick was devised in 1975 to predict UV phototherapy response, and types I-IV are defined by self-reported BURN AND TAN BEHAVIOUR, not by colour. Types V and VI were added later — two categories for the majority of the world's skin. Dermatology AI adopted it because it was the label that happened to be in the datasets, not because it measures what we need. The Monk Skin Tone scale is the better instrument and this tool accepts it; we use Fitzpatrick here only because it is what the available data carries.