Benchmarks

subjects

sub-00048

sub-00048: measured, but not by the route the primary figure accepts: its expert mask names an acquisition the dataset does not ship, and the moving image was identified by grid identity instead. The comparison is published in the sensitivity analysis rather than folded into the primary.

detected

16

of

26

value

0.6154

a subject counts as detected when at least one predicted cluster overlaps the expert mask by one voxel or more. Reference standard: the expert lesion mask -- the radiologist-drawn *_FLAIR_roi.nii.gz, carried into the prediction's space through a rigid FLAIR->T1 registration recomputed locally from the two public images. Denominator 26 subjects.

of

26

value

0.6154

the harsher question: the largest predicted cluster, which is what a reader looks at first, overlaps the expert mask. Reference standard: the expert lesion mask -- the radiologist-drawn *_FLAIR_roi.nii.gz, carried into the prediction's space through a rigid FLAIR->T1 registration recomputed locally from the two public images. Denominator 26 subjects.

clusters hitting lesion

16

clusters total

26

value

0.6154

A predicted cluster counts as a true positive when it overlaps the expert mask by one voxel or more. A cluster is a connected component of at least 10 voxels: MELD works on a surface mesh, and the map back into the volume leaves one-voxel specks that are not places a reader gets sent to. The threshold is not tuned and not delicate: the cohort's components run 1, 1, ... 5, 6 and then nothing until 204, so any minimum from 7 to 204 yields the same 28 clusters. The discarded voxels are still counted against the mask. Reference standard: the expert lesion mask -- the radiologist-drawn *_FLAIR_roi.nii.gz, carried into the prediction's space through a rigid FLAIR->T1 registration recomputed locally from the two public images. Denominator 26 subjects.

mean

0.385

median

0

How many extra places per subject a reader would be sent for nothing. Same cluster definition as the row above: a connected component of at least 10 voxels. Counting specks instead would have reported 46 candidates across the cohort where MELD's own tables list 28, and all eighteen extras -- every one of them six voxels or smaller -- would have landed in this column. Reference standard: the expert lesion mask -- the radiologist-drawn *_FLAIR_roi.nii.gz, carried into the prediction's space through a rigid FLAIR->T1 registration recomputed locally from the two public images. Denominator 26 subjects.

mean

0.2176

median

0.1604

Median is the headline because the distribution is bimodal: subjects the model localises score 0.13 to 0.60, and subjects it misses score exactly zero. A mean over those two populations describes neither. Reference standard: the expert lesion mask -- the radiologist-drawn *_FLAIR_roi.nii.gz, carried into the prediction's space through a rigid FLAIR->T1 registration recomputed locally from the two public images. Denominator 26 subjects.

mean

0.1394

median

0.0873

Reference standard: the expert lesion mask -- the radiologist-drawn *_FLAIR_roi.nii.gz, carried into the prediction's space through a rigid FLAIR->T1 registration recomputed locally from the two public images. Denominator 26 subjects.

mean

0.3357

median

0.3035

The fraction of expert-mask voxels the prediction covers. Not the same quantity as lesion-level sensitivity, and much lower: MELD localises a lesion far more often than it outlines one. Reference standard: the expert lesion mask -- the radiologist-drawn *_FLAIR_roi.nii.gz, carried into the prediction's space through a rigid FLAIR->T1 registration recomputed locally from the two public images. Denominator 26 subjects.

mean

0.2731

median

0.1463

Reference standard: the expert lesion mask -- the radiologist-drawn *_FLAIR_roi.nii.gz, carried into the prediction's space through a rigid FLAIR->T1 registration recomputed locally from the two public images. Denominator 26 subjects.

mean

26.7

median

6.8

Distance between the centre of mass of all predicted voxels and the centre of mass of the expert mask. The mean is four times the median because a subject whose prediction lands in the wrong hemisphere contributes 70 to 110 mm; that is a real miss, not an outlier to be trimmed. Reference standard: the expert lesion mask -- the radiologist-drawn *_FLAIR_roi.nii.gz, carried into the prediction's space through a rigid FLAIR->T1 registration recomputed locally from the two public images. Denominator 26 subjects.

of

23

value

0.8261

the sign of the x coordinate of the predicted centroid, against the hemisphere recorded in participants.tsv. Reference standard: the recorded hemisphere and lobe -- the hemisphere and lobe recorded in ds004199's participants.tsv, anchored in surgery and histopathology. Coarser than the mask, and available for subjects whose mask cannot be registered.

subjects

sub-00125, sub-00095, sub-00100

An empty candidate set is a model that produced nothing to look at. It is not a normal MRI and must never be rendered as one: every subject in this cohort has a surgically or histologically anchored lesion, so an empty prediction is a miss by construction.

adds

sub-00048

detected

17

of

27

value

0.6296

NOT the primary figure. The named acquisition is genuinely absent, but a shipped image carries the mask's exact shape and its affine to 3.2e-4, and its sidecar SeriesDescription names the missing acquisition's own coil. A mask cannot be drawn on a grid nobody was looking at, so the shipped image is that scan under a different `acq` label. Registered through it the case is a hit, which a world-coordinate overlay would have reported as a miss. The evidence is a grid match rather than the filename the rest of the cohort resolves by, and the admissibility policy that produced the primary figure should not be relaxed for the one case it excludes. Published beside it, not instead of it. Reported because a result that moves from 0.615 to 0.630 when the contested case is admitted has had its robustness demonstrated; one reported alone has had it asserted.

median

0.1408

primary median

0.1604

The contested case is a hit with a low Dice, so admitting it raises sensitivity and lowers the median overlap. Both move, in opposite directions, and neither movement is large.

of

24

primary

0.8261

primary of

23

value

0.8333

Reference standard: Reference standard: the recorded hemisphere and lobe -- the hemisphere and lobe recorded in ds004199's participants.tsv, anchored in surgery and histopathology. Coarser than the mask, and available for subjects whose mask cannot be registered.

false positive clusters per patient range

0.92308, 0.38462

lesion sensitivity unchanged

true

plateau begins at

7

published floor

10

swept

1, 2, 3, 5, 7, 10, 20, 50, 100

the floor changes false-positive clusters per patient and cluster PPV, and nothing else. Lesion sensitivity is identical at every value from 1 to 100 -- no discarded component was ever the one that found a lesion -- and Dice, centroid distance and hemisphere agreement are computed from the voxel mask rather than the cluster decomposition, so the floor cannot reach them at all. The published constant sits on the plateau rather than on a slope, which is what makes it a measured choice rather than an arbitrary one.

from a training contributing site

ALL

single site

true

sites

university_hospital_bonn

every evaluated subject comes from a site that contributed to the cohort MELD Graph declares, so no subset of this result is independent and no further case from this dataset could make it so. The tier is OVERLAPPING_DATASET_EVALUATION at its strongest reading, not its weakest. STILL UNKNOWN: whether these particular scans were in MELD's training data. Bonn contributing to that cohort is a fact about the institution; ds004199 is a separately collected series. Subject-level overlap is possible, is not demonstrated, and cannot be excluded from anything recorded here -- it would need MELD's own subject manifest, which is not published.

max case dice delta

0.0027

median dice delta

0

runs

2

verdicts changed

0

The FLAIR->T1 fit is stochastic, so the same cohort measured twice does not return bit-identical floats. Measured rather than assumed: every detected/missed verdict, every denominator and every cluster count was identical across two full runs, and the largest movement in any case Dice was 0.0027. Numbers below the third decimal are fit noise and should not be read.

recomputed locally

27

required from container

0

An earlier pass reported this as unrecoverable, on the grounds that the transform was written inside the pipeline container and never exported. That was a misreading of what the transform is. The prediction comes back on the raw T1's own grid -- identical shape and affine, checked per subject -- so the transform needed is FLAIR->T1, a function of two images the public dataset ships. Recomputed locally it reproduces the one exported case to Dice 0.9928, and nothing had to survive the container.

excluded subjects

0

not observed

22

observed cases

6

max seconds

7.9 h

median seconds

5.3 h

min seconds

3.6 h

n

28

reasons

university_hospital_bonn contributed to both ds004199 and the training cohort meld_graph declares, recorded from https://meldproject.github.io/docs/collaborator_list_180725.pdf

shared sites

university_hospital_bonn

status

OVERLAPS