11: SynthSR: A public AI tool to turn heterogeneous clinical brain scans into high-resolution T1-weighted images for 3D morphometry

Blog post written by: Katherina Terefenko

Based on: Iglesias, J. E., Billot, B., Balbastre, et al. (2023). SynthSR: A public AI tool to turn heterogeneous clinical brain scans into high-resolution T1-weighted images for 3D morphometry. Science Advances, 9(5). https://doi.org/10.1126/sciadv.add3607

Introduction: The Untapped Treasure in Clinical Brain Scans

Radiology departments are filled with a wide range of brain MRI scans, captured with different modalities, orientations, resolutions, and contrasts. Each scan is tailored to the patient's needs. While this flexibility suits clinical care, it creates significant variation in scan quality and format. 
Most neuroimaging tools, like FreeSurfer or FSL, require high-resolution, isotropic T1-weighted scans¹, a standard that many clinical MRIs do not meet. As a result, millions of brain scans go unused in research.
SynthSR adresses this gap. This open-source AI tool converts heterogeneous clinical scans into standardized 1-mm isotropic T1-weighted images, making them compatible with tools for morphometry, segmentation, and registration. It also includes an especially ambitious feature: lesion inpainting, which enables segmentation even when pathology is present.

The Challenge of Diverse MRI Modalities

Figure 3 - Isotropy vs. Anisotropy in Brain Scans: Isotropic voxels are cube-shaped and ideal for 3D analysis. Many clinical scans are not, which makes SynthSR’s standardization step essential.Figure 2 - T1, T2, and FLAIR - Different MRI Contrasts: T1 scans show anatomy clearly, T2 highlights fluid, and FLAIR suppresses it to reveal lesions. SynthSR can convert any of these into a usable T1-weighted image. Figure 1 - Understanding MRI Scan Planes: Axial, coronal, and sagittal planes slice the brain in different directions: horizontal, frontal, and side views. SynthSR can handle scans in any of these orientations. Magnetic resonance imaging (MRI) scans vary along several key dimensions: orientation, contrast, resolution, and isotropy. 
These settings are important for clinical care but introduce inconsistencies that complicate research analysis.

Orientation refers to the scan plane - axial, sagittal, or coronal - chosen based on diagnostic need (Figure 1).

Contrast depends on the timing of radiofrequency pulses and signal measurements. Common image types include:

• T1-weighted images: highlights fat and anatomical detail
• T2-weighted images: emphasizes fluids
• FLAIR: suppresses fluid signals to reveal lesions

Figure 2 compares how tissues appear in common contrasts.

Resolution and isotropy further complicate things. Clinical brain scans often use thicker slices to save time—typically around 5 mm.
Some scans are isotropic, with equal voxel dimensions in all directions, but many are anisotropic, which makes 3D analysis difficult.
High-resolution scans used in research usually have 1 mm isotropic resolution.
Figure 3 illustrates different resolutions and shows a visualization of isotropy.

SynthSR’s aim is to overcome these inconsistencies by converting scans of various resolutions and orientations into standardized, 1 mm isotropic T1-weighted images, ready for morphometric analysis.

Standardizing the Brain and Repairing Its Gaps: How SynthSR Works

At its core, SynthSR takes any brain MRI scan, regardless of modality, orientation, or resolution, and produces a synthetic 1-mm isotropic T1-weighted MPRAGE image. This output format is compatible with virtually all standard neuroimaging tools and enables downstream tasks like morphometry, segmentation, and registration. SynthSR is designed for ease of use: it runs “out of the box” as part of the open-source FreeSurfer software suite, with no need for retraining or parameter tuning.

In addition, SynthSR includes a built-in inpainting step. In cases where the input scan contains lesions, such as those caused by tumors, strokes, or other pathologies, the model generates anatomically plausible tissue to fill in those areas. This allows standard tools, which often fail when encountering abnormal anatomy, to process the image as if it were healthy, enabling segmentation and registration in otherwise unusable scans.

The Brains of the Operation: Two U-Nets Working Together

Figure 4 - The SynthSR Pipeline at a Glance: From raw clinical scan to high-resolution output, SynthSR creates synthetic T1 images ready for segmentation. It works out of the box and requires no retraining. SynthSR’s architecture is based on two interconnected 3D convolutional neural networks known as U-Nets.

A U-Net is a type of CNN widely used in medical imaging tasks. Its architecture resembles the letter "U", with a downsampling path that captures contextual information and an upsampling path that reconstructs detailed spatial features. They are commonly used in medical
imaging tasks that require high-resolution outputs, like segmentation or image synthesis. Figure 4 illustrates the overall structure of a U-Net.

In SynthSR:

• The first U-Net acts as a regression network that generates the synthetic MPRAGE image from the clinical input scan.
• The second U-Net functions as a segmentation network. It is pre-trained and kept frozen during SynthSR's training.
  Its role is to check whether the output image is suitable for anatomical segmentation.

In other words, one network learns to reconstruct plausible brain anatomy, while the other ensures that the reconstruction retains enough anatomical structure to be correctly segmented
by neuroimaging tools.

This dual-network setup ensures that the outputs are not just realistic but usable. SynthSR doesn’t just generate images that look correct. It generates images that are informative, consistent, and compatible with analysis pipelines. Figure 5 illustrates this process.

But generating standardized images is only part of the challenge. SynthSR must also handle cases where parts of the brain are structurally missing or visibly abnormal.

Figure 5 - U-Net in Action, SynthSR’s Neural Backbone: A pair of U-Nets powers SynthSR: one generates the image, the other checks if it can be segmented. This structure helps balance realism with usability.

Inpainting Lesions: Softened, but not Forgotten

This brings us to SynthSR’s most ambitious feature: lesion inpainting.

Figure 6 - Inpainting Makes the Broken Usable: SynthSR fills in damaged regions with realistic-looking tissue. That way, standard segmentation tools can still process the image. Brain lesions caused by tumors, strokes, or other conditions often disrupt neuroimaging workflows. Standard tools may fail entirely or produce unreliable results when abnormal anatomy is present.

Machine Learning models are particularly prone to two extremes:

• When trained mostly on healthy data, they may erase pathology entirely, smoothing over lesions as if they were never there
• When trained too heavily on pathological cases, they may hallucinate lesions in healthy tissue

SynthSR takes a more balanced approach by using inpainting to fill in damaged areas with anatomically plausible, healthy-looking tissue. The goal is not to remove the lesion, but to adjust its appearance so that tools like FreeSurfer or FSL,
which assume healthy anatomy, can process the scan reliably. Figure 6 visualizes this step: a lesioned input scan is filled in with plausible anatomy, enabling reliable segmentation downstream.

Critically, SynthSR preserves the lesion’s spatial structure. While the intensity values are modified to reduce interference with morphometric analysis, the shape, location, size, and boundaries of the lesion remain intact. This allows researchers to continue
studying the lesion’s geometry or impact after segmentation and registration are complete.

Inpainting is applied only to the SynthSR-generated image; the original scan remains untouched. Once segmentation, labeling, or surface reconstruction is done on the synthetic image, these transformations can be applied back to the original. This ensures
that lesions are never truly lost—they are just temporarily softened so the tools can do their work.

While inpainting is not unique to SynthSR, the way it is integrated into an out-of-the-box pipeline is both technically elegant and practically ambitious.
But it also raises an important question: just how reliable are the downstream analyses when part of the input is synthetic?

Can We Trust Synthesized Brains with Lesions? SynthSR in Stroke and Tumor Cases

SynthSR’s inpainting step is designed to make scans with structural abnormalities usable in existing neuroimaging workflows. But how well does this actually work when real pathology is present? To explore this, the authors applied SynthSR to two types of clinical cases where lesions are common and often disruptive: stroke and glioma.

These cases present different challenges, but raise the same question: can we trust imaging results when part of the brain has been filled in?

Figure 7 - Registering Glioma Scans, Before and After SynthSR: Brain tumors distort anatomy and break registration tools. SynthSR smooths those disruptions, enabling cleaner, more stable alignments.

Glioma Scans: Aligning the Unalignable

One promising use of SynthSR is improving image registration for brain tumor scans. In studies tracking glioma progression or treatment response, researchers often need to align pre- and post-operative MRI scans.
Tumors distort anatomy, making standard registration unreliable, especially across timepoints or scan types.

To address this, the researchers applied SynthSR to pre- and post-operative MRI scans from glioma patients in various contrasts. For each case, SynthSR generated a standardized, inpainted 1-mm T1-weighted image, while temporarily smoothing
over the tumor region to allow registration algorithms to focus on the surrounding anatomy. This process is visualized in Figure 7, which shows how SynthSR helps align anatomical structures more cleanly across time points.

The result: clearer, more stable alignments. Compared to registration on the original scans, SynthSR’s output consistently reduced landmark errors across all image contrasts and alignment settings by 10 to 20 percent. In particular, it avoided a common failure mode:
misaligning tumor edges between timepoints, which previously led to tangled deformation fields. Even in aggressive registration setups, SynthSR’s standardized outputs remained more robust.

SynthSR’s ability to produce clean, standardized images even in the presence of large tumors addresses a longstanding bottleneck in neuro-oncology: accurate longitudinal alignment. Tumors distort not just anatomy but also the assumptions baked into registration algorithms.
By smoothing over these distortions SynthSR allows registration tools to operate with a new level of consistency. This doesn’t just improve alignment metrics; it opens the door to tracking tumor progression more reliably across time and imaging modalities.

Stroke Scans: Filling in the Blanks

Figure 8 - Stroke Scan, Repaired and Segmented: Stroke lesions often block segmentation entirely. SynthSR restores continuity so the brain can still be analyzed using standard tools. If glioma scans challenge SynthSR with distortion, stroke scans push it further—adding not just misalignment, but actual missing tissue. Lesions caused by strokes can wipe out large sections of brain tissue, leaving holes that confuse standard segmentation tools or cause them to fail entirely.
This makes accurate segmentation especially important for studying structural changes and predicting outcomes in stroke research².

To test the limits of its inpainting approach, the authors applied SynthSR to the publicly available ATLAS dataset, which includes MRI scans from stroke patients along with manually segmented lesion masks.
SynthSR filled in the damaged areas with anatomically plausible tissue, enabling segmentation pipelines like FreeSurfer to operate on scans that would otherwise be unusable.
Figure 8 illustrates this process, showing how SynthSR restores continuity to a lesioned brain scan and unlocks meaningful anatomical segmentation.

The key question was whether SynthSR’s segmentations, which are based on inpainted scans, could still support meaningful morphometric analysis despite the original tissue loss.
To answer this question, the team extracted volumes from the segmentations of several brain regions and compared them between the lesioned and non-lesioned hemispheres. They observed strong asymmetries in some regions (thalamus, putamen), while other regions, like the hippocampus,
showed little or no asymmetry, meaning a stroke had the same effect on the region’s volume regardless of the hemisphere it occurred on.
The (a)symmetry patterns along with further observations the research team made based on segmentations of the synthesized T1 images were broadly consistent with findings from prior clinical studies.
These findings weren’t the focus—but they serve as indirect validation of the segmentation quality SynthSR enables.

This capacity for plausible inpainting also enabled another key application: building a population-level lesion atlas. By co-registering SynthSR-generated scans, the team was able to construct spatial probability maps of stroke damage across patients.
The resulting atlas revealed familiar lesion patterns in the basal ganglia and deep white matter—insights that have been stated in prior studies but are generally difficult to achieve when registration fails due to anatomical voids.
Figure 9 shows slices from the median brain template and the corresponding lesion distribution map made possible by SynthSR’s ability to inpaint and align damaged scans across patients.

Figure 9 - Stroke Lesion Distribution Across the Brain: By enabling alignment, SynthSR made it possible to build this population-level lesion atlas. Stroke damage is most common in deep white matter and the basal ganglia and appears to be more severe on the right hemisphere.

Taken together, the glioma and stroke studies are encouraging but they also highlight a deeper tension between utility and authenticity in neuroimaging workflowsStroke imaging is messy by nature - lesions erase anatomy, confuse segmentation algorithms, and often lead to exclusion from research datasets. SynthSR flips that logic: it enables inclusion by filling in what’s missing. This makes stroke scans analyzable, but it also means researchers must think carefully about what they’re analyzing. Are they studying a patient’s real anatomy, or a best-guess reconstruction of it? SynthSR’s promise is not in perfect fidelity, but in allowing large-scale analysis of groups that were systematically left out. That shift could profoundly impact how we understand post-stroke outcomes, if handled with care.

Critical Reflection - Not Perfect, but Transformative

Ultimately, the question remains: when we fill in damaged brain regions with synthetic tissue to make a scan analyzable, how confident can we be in the insights we draw from it?

This is where SynthSR walks a fine line. Its inpainting step allows scans with abnormal anatomy to be processed by tools that assume healthy structure, but that assumption is inherently fragile. SynthSR doesn’t aim to recreate pathology with clinical fidelity; it aims to make the image usable. That distinction is crucial. In studies that focus directly on the pathology, such as tracking lesion progression or modeling tumor growth, these synthetic images may not be suitable for primary analysis. Researchers should be cautious, validate results carefully, and, where possible, compare outputs from both the original and the inpainted scans.

There are other limitations too. Synthesizing a high-resolution T1 scan from a different contrast or low-resolution input inevitably involves interpolation. Some structural details simply aren’t present in the source data and can’t be recovered by even the best model. The quality of the output depends on the quality of the input, and not all clinical scans are created equal.

Still, SynthSR’s strength lies not in anatomical precision but in practical access. It makes it possible to include scans that would otherwise be excluded, to work with patient groups often left out of research, and to analyze patterns across larger, more diverse datasets. It shifts the focus from exclusion to inclusion, while reminding us that inclusion must be handled with care.

Why It Matters - And Where the Limits Lie

SynthSR is built around a clear goal: transforming messy, variable clinical scans into standardized T1-weighted images that work with established research tools. This standardization is not just a technical convenience. It helps overcome structural limitations in neuroimaging, such as restricted access to large, diverse, and standardized datasets, which are known barriers to reproducibility and generalizability³.

But SynthSR doesn’t stop at formatting. Its inpainting step goes further, making it possible to analyze scans that would otherwise be unusable due to lesions or structural damage. This makes the tool both ambitious and easy to overinterpret. It doesn't promise anatomical fidelity in every detail, especially when pathology is involved. It promises usability and access. Access to data, to populations, and to questions that were previously unreachable.

When used with that awareness, SynthSR doesn’t just expand what we can analyze. It broadens how we think about inclusion in brain imaging research, while reminding us that inclusion, too, must be critically examined.

The future of neuroimaging may not lie in perfect scans, but in smart tools that can make sense of imperfect ones. SynthSR is a bold step in that direction.

References

¹Martinos Center for Biomedical Imaging. (2020). FreeSurfer Beginner's Guide. https://surfer.nmr.mgh.harvard.edu/fswiki/FreeSurferBeginnersGuide

²Liew, S.-L., Anglin, J. M., Banks, N. W., Sondag, M., Ito, K. L., Kim, H., Chan, J., Ito, J., Jung, C., Khoshab, N., Lefebvre, S., Nakamura, W., Saldana, D., Schmiesing, A., Tran, C., Vo, D., Ard, T., Heydari, P., Kim, B., ... Stroud, A. (2018). Data descriptor: A large, open-source dataset of stroke anatomical brain images and manual lesion segmentations. Scientific Data, 5, 180011. https://doi.org/10.1038/sdata.2018.11

³Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J.-B., Vul, E., & Yarkoni, T. (2017). Scanning the horizon: Towards transparent and reproducible neuroimaging research. Nature Reviews Neuroscience, 18(2), 115–126. https://doi.org/10.1038/nrn.2016.167

Figures 1 - 5: Generated with Sora, modified and extended manually.

Figures 6 - 9: Adapted from Iglesias, J. E., Billot, B., Balbastre, et al. (2023). SynthSR: A public AI tool to turn heterogeneous clinical brain scans into high-resolution T1-weighted images for 3D morphometry. Science Advances, 9(5). https://doi.org/10.1126/sciadv.add3607

ChatGPT Key Prompts

• I have created a rough outline for a blog post on this paper <paper; outline>. Those are the contents I want to convey. Can you help to restructure my outline in a suited way for the blog post? I am trying to highlight/focus on these <parts> of the paper.
• I would like to explain <machine learning terminology> in mostly non-technical language. How can I phrase it so it’s easily understandable?
• Can you shorten this paragraph and thereby make it more concise?
• I have written a blog post on this paper <paper, blog post>. Can you review my blog post regarding grammar and language and provide critical feedback?