OcuWeave
Evidence-Grounded Vision–Language Learning with Incomplete Clinical Records for Ophthalmic Disease Assessment
Data Sorcerers ·
Manuscript in preparation.
Abstract
OcuWeave is a research project on lesion-level ophthalmic assessment from retinal images and incomplete clinical records. Ocular toxoplasmosis is the primary disease focus. We propose an independently implemented network that combines compact visual representations, query-based mask prediction, and explicit handling of unavailable clinical evidence, without a U-Net backbone. The initial experiment targets visible lesion segmentation; image–record fusion requires genuine patient-linked data, while longitudinal assessment and visual-function prognosis are separate extensions. The evaluation plan covers segmentation quality, missing-information ablations, calibration, and computational cost. This page describes the research protocol and work in progress. Training results, comparative performance, and clinical utility have not yet been established.
Research protocol · Updated . No trained model or manuscript is released on this page.
Overview
OcuWeave investigates how retinal images, clinical notes, and structured records can support ophthalmic disease assessment when clinical evidence is incomplete. The research direction combines visual grounding, explicit missingness, and uncertainty estimation, with ocular toxoplasmosis as a disease-specific focus. The first experimental target is a pixel-level map of visible toxoplasmosis lesions, building on the imaging problem and benchmark established in earlier work [1], [2].
The central question is whether a model can preserve reliable visual localization when clinical information is missing, incomplete, or inconsistent. Combining an image encoder with a language model is not, by itself, the intended contribution. The proposed contribution must be demonstrated through controlled comparisons and evidence-specific tests.
Primary focus
Ocular toxoplasmosis lesion segmentation. Lesion extent, activity, and functional outcome remain distinct targets.
Research hypothesis
Explicit evidence availability and region-level reasoning may improve robustness under incomplete clinical records.
Open research
Release reproducible code, configurations, and permitted artifacts. Document unsuccessful experiments and data limitations.
Introduction
An ophthalmic image captures only part of the clinical picture. Lesion appearance must be interpreted alongside the available history, examination, treatment, and follow-up information. The ocular toxoplasmosis common-data-elements consensus provides a reference for structuring these observations; it is a consensus specification, not a ready-to-train patient dataset [3].
Retinal foundation models such as RETFound and language–image approaches such as FLAIR show why representation learning and clinical semantics are relevant starting points [4], [5]. OcuWeave does not claim to introduce ophthalmic vision–language learning. It investigates a narrower problem: lesion localization when the completeness and reliability of the accompanying evidence vary.
Three distinctions guide the study. A lesion mask is not a diagnosis of active infection. A change in lesion area is not automatically a change in visual function. A clinical note generated from a label is not an independent medical record. These distinctions determine which inputs, targets, and claims each experiment can support.
Methodology
Data acquisition and cohort definition
Each dataset has a separate task and access audit. Harvard-FairVLMed supports paired ophthalmic image–text research, with a distinction between original de-identified notes and generated summaries [6]. BRSET contributes fundus images with structured clinical and demographic information [7]. OLIVES provides longitudinal ophthalmic imaging and clinical labels for DR/DME-related studies [8]. GRAPE offers a separate setting for glaucoma follow-up and visual-field analysis [9].
| Dataset | Planned role | Key boundary |
|---|---|---|
| OTFID v3 [1] | Primary toxoplasmosis image and lesion-mask experiment. | Paired narrative EMR and patient-level linkage remain unverified. |
| FairVLMed [6] | Separate image–text and missing-note experiments. | Glaucoma domain; audit diagnosis leakage. Non-commercial research terms apply. |
| BRSET [7] | Fundus plus structured clinical variables. | Credentialed access; structured fields are not full narrative EMR. |
| OLIVES [8] | Longitudinal imaging and biomarker studies. | DR/DME setting; not a toxoplasmosis follow-up cohort. |
| GRAPE [9] | Glaucoma follow-up and compatible visual-field endpoints. | Progression labels do not establish a time-to-blindness endpoint. |
Cleaning will check archive integrity, image–mask alignment, unreadable files, duplicates, missing fields, and label contradictions. Original data will be preserved. Patients, eyes, and visits will be grouped before splitting when verified identifiers exist. If they do not, any image-level split will be explicitly reported as exploratory. Records from unrelated patients will never be paired to manufacture a multimodal cohort.
Network design without U-Net
The proposed network uses a compact visual encoder and a query-based lesion-mask head. Convolutional-attention representations and mask-query decoding are design references, rather than claims of invention [10], [11]. The main network is independently implemented without a U-Net backbone or a symmetric U-Net decoder. Its final configuration remains subject to ablation and profiling.
- Retinal imageAudited input and geometry
- Visual encoderCompact multi-scale features
- Lesion queriesRegion-to-feature interaction
- Lesion maskPixel-level prediction
Planned extension: patient-linked clinical evidence → availability-aware fusion → lesion queries.
Clinical features and language representations will carry explicit availability indicators. Unknown findings must not silently become negative findings. Missing-modality methods such as PASSION inform the comparison, while the difference between missing imaging modalities and missing clinical records must be tested directly [12]. Ground-truth masks and target diagnoses will remain separate from model inputs.
Training and evaluation protocol
The first run is planned in Google Colab, beginning with image-only binary lesion segmentation. Original network code will use PyTorch for differentiation; isolated C++ operations will have explicit interfaces and numerical parity checks. Implementing a network from scratch does not imply training a large language model from random initialization.
Comparisons will use fixed splits, matched training budgets, and validation-only model selection. The initial segmentation objective combines binary cross-entropy and soft Dice. Planned evaluation includes Dice and IoU, lesion sensitivity, boundary behavior, small lesions, and empty-mask cases. Metric definitions and aggregation units will be specified before interpreting results [13].
Subsequent experiments will test missing, irrelevant, and conflicting clinical information; calibration and selective prediction; and compatible external cohorts. Longitudinal models may use only information available at the prediction time. Reporting will document data provenance, patient linkage, uncertainty, compute, and failure cases in line with TRIPOD+AI where applicable [14].
Use cases
These are intended research scenarios, not clinically validated capabilities.
Map visible toxoplasmosis lesions
Outline annotated lesions for inspection and measurement. “Disease origin” or an unseen early lesion cannot be inferred solely from a segmentation mask.
Compare follow-up examinations
Study registered images from the same eye to quantify structural change. A clinician-defined endpoint is needed to interpret improvement or deterioration.
Study glaucoma and diabetic retinal disease
Use separate cohorts for visual-field progression or DR/DME-related imaging tasks. Visual-function forecasting requires dated follow-up and an explicit outcome definition.
Evaluate incomplete clinical context
Test whether predictions remain stable when records are missing and whether contradictory information causes identifiable failures.
Cardiovascular disease, stroke, COPD, diabetes mellitus, and cancer are possible later methodological extensions. No ability to diagnose these conditions from retinal images is claimed for OcuWeave.
Results and discussion
No model-performance results are available yet. This section records the current research evidence and the questions that must be resolved before a benchmark can be interpreted.
- Literature curation
- A collection of 117 publications has been assembled: 113 from 2022–2026 and four foundational exceptions, including two Indonesian-journal papers. This is metadata/abstract screening with selected full-text checks, not a completed systematic review.
- Dataset readiness
- Public dataset sources and access conditions have been identified. Release-level cleaning, patient linkage, and task compatibility are under audit. A public toxoplasmosis cohort combining masks, paired narrative EMR, and longitudinal outcomes has not been verified.
- Implementation and training
- The released software is version 0.1.1; the next pipeline is in development. Colab is the planned training environment. No completed training run, held-out score, or clinical evaluation is reported.
The main research risk is confounding a richer input with better reasoning. Notes may reveal the target diagnosis; duplicate or related images may cross an unverified split; and uncertainty estimates may fail under a new acquisition setting. A useful result must survive these checks, not only improve an average segmentation score.
Positive and negative findings will be reported against matched baselines, with the exact cohort, checkpoint, code revision, and evaluation protocol. Ablations must establish whether any gain comes from the proposed evidence mechanism rather than extra parameters, data, or privileged labels. Novelty and clinical value remain hypotheses.
Conclusion
OcuWeave sets out to study lesion-level ophthalmic assessment when clinical evidence is incomplete. Its immediate priority is a reproducible toxoplasmosis segmentation experiment with an independently implemented, non-U-Net network. Multimodal grounding, longitudinal comparison, and prognosis will be added only when the data and supervision support them.
The next milestones are to complete the dataset audit, freeze the evaluation protocol, run image-only baselines in Colab, and evaluate the proposed components before making performance or publication claims.
Code: MIT license. Dataset and future checkpoint rights are assessed separately. Software changelog · Software citation metadata. A paper citation and downloadable manuscript will be added when available.
References
Selected sources from the research collection, numbered in order of first citation. These are prior publications, not OcuWeave results.
O. Cardozo et al., “Dataset of fundus images for the diagnosis of ocular toxoplasmosis,” Data in Brief, vol. 48, Art. no. 109056, 2023. doi: 10.1016/j.dib.2023.109056.
S. S. Alam, S. B. Shuvo, S. N. Ali, F. Ahmed, A. Chakma, and Y. M. Jang, “Benchmarking deep learning frameworks for automated diagnosis of ocular toxoplasmosis: A comprehensive approach to classification and segmentation,” IEEE Access, vol. 12, pp. 22759–22777, 2024. doi: 10.1109/ACCESS.2024.3359701.
W. Rojas-Carabali et al., “Common data elements for observational studies in ocular toxoplasmosis: A Delphi consensus,” Journal of Ophthalmic Inflammation and Infection, vol. 15, Art. no. 68, 2025. doi: 10.1186/s12348-025-00525-2.
Y. Zhou et al., “A foundation model for generalizable disease detection from retinal images,” Nature, vol. 622, pp. 156–163, 2023. doi: 10.1038/s41586-023-06555-x.
J. Silva-Rodríguez, H. Chakor, R. Kobbi, J. Dolz, and I. Ben Ayed, “A Foundation Language-Image Model of the Retina (FLAIR): Encoding expert knowledge in text supervision,” Medical Image Analysis, vol. 99, Art. no. 103357, 2025. doi: 10.1016/j.media.2024.103357.
Y. Luo et al., “FairCLIP: Harnessing fairness in vision-language learning,” Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 12289–12301. Official proceedings ↗.
L. F. Nakayama et al., “BRSET: A Brazilian multilabel ophthalmological dataset of retina fundus photos,” PLOS Digital Health, vol. 3, no. 7, Art. no. e0000454, 2024. doi: 10.1371/journal.pdig.0000454.
M. Prabhushankar, K. Kokilepersaud, Y.-y. Logan, S. Trejo Corona, G. AlRegib, and C. Wykoff, “OLIVES dataset: Ophthalmic labels for investigating visual eye semantics,” Advances in Neural Information Processing Systems, 2022. Official proceedings ↗.
X. Huang et al., “GRAPE: A multi-modal dataset of longitudinal follow-up visual field and fundus images for glaucoma management,” Scientific Data, vol. 10, Art. no. 520, 2023. doi: 10.1038/s41597-023-02424-4.
M.-H. Guo, C.-Z. Lu, Q. Hou, Z. Liu, M.-M. Cheng, and S.-M. Hu, “SegNeXt: Rethinking convolutional attention design for semantic segmentation,” Advances in Neural Information Processing Systems, 2022. Official proceedings ↗.
B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar, “Masked-attention mask transformer for universal image segmentation,” Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 1290–1299. Official proceedings ↗.
J. Shi, C. Shang, Z. Sun, L. Yu, X. Yang, and Z. Yan, “PASSION: Towards effective incomplete multi-modal medical image segmentation with imbalanced missing rates,” Proc. 32nd ACM International Conference on Multimedia, 2024, pp. 456–465. doi: 10.1145/3664647.3681543.
L. Maier-Hein et al., “Metrics reloaded: Recommendations for image analysis validation,” Nature Methods, vol. 21, pp. 195–212, 2024. doi: 10.1038/s41592-023-02151-z.
G. S. Collins et al., “TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods,” BMJ, vol. 385, Art. no. e078378, 2024. doi: 10.1136/bmj-2023-078378.