Representation Learning

CONCH (CONtrastive learning from Captions for Histopathology) is a vision-language foundation model pretrained on 1.17 million histopathology image-caption pairs. It achieves state-of-the-art performance across 14 diverse benchmarks including histology image classification, segmentation, captioning, text-to-image and image-to-text retrieval. As a multimodal model bridging visual pathology data with biomedical text, CONCH enables zero-shot transfer and minimal fine-tuning for diverse computational pathology tasks.

The accelerated adoption of digital pathology and advances in deep learning have enabled the development of robust models for various pathology tasks. However, model training is often difficult due to label scarcity. Additionally, most models in histopathology leverage only image data. We introduce CONCH, a visual-language foundation model developed using diverse sources of histopathology images, biomedical text, and over 1.17 million image-caption pairs via task-agnostic pretraining. Evaluated on 14 diverse benchmarks, CONCH achieves state-of-the-art performance on histology image classification, segmentation, captioning, and cross-modal retrieval.

transferGWAS performs GWAS directly on full medical images using deep transfer learning: (1) a pretrained CNN (ResNet-based architecture, pretrained on ImageNet) extracts feature embeddings from raw images; (2) these learned representations are used as quantitative phenotypes for genetic association testing. Applied to UK Biobank retinal fundus images, identified 60 genomic regions including 7 novel candidate loci for eye-related traits. First demonstration of direct GWAS on whole images without predefined phenotype engineering.

deep transfer learning, pretrained CNN, ResNet, retinal fundus, whole-image GWAS, representation learning, UK Biobank

MOTIVATION: Medical images can provide rich information about diseases and their biology. However, investigating their association with genetic variation requires non-standard methods. We propose transferGWAS, a novel approach to perform genome-wide association studies directly on full medical images. First, we learn semantically meaningful representations of the images based on a transfer learning task, during which a deep neural network is trained on independent but similar data. Then, we perform genetic association tests with these representations. RESULTS: We validate the type I error rates and power of transferGWAS in simulation studies of synthetic images. Then we apply transferGWAS in a genome-wide association study of retinal fundus images from the UK Biobank. This first-of-a-kind GWAS of full imaging data yielded 60 genomic regions associated with retinal fundus images, of which 7 are novel candidate loci for eye-related traits and diseases.