Abstract

Detection, segmentation, and classification of cells form an integral and critical step in most biomedical imaging experiments. Deep learning methods have made impressive progress in recent years, including Vision Transformers (ViTs) for their ability to generalize across image modalities and include more context than convolutional neural networks (CNNs). Although transformer-based architectures gained traction in cell segmentation (Israel et al, 2023; Archit et al, 2023), the pace at which biologists generate ground-truth microscopy images has not changed. In contrast to the field of computer vision applied to natural images, the large amount of annotated data necessary to train vision transformers has created a lag in biomedical research (Zhu et al, 2023; Kirillov et al, 2023). To close this gap, we are developing a new framework for the large-scale synthesis of 2D and 3D multimodal training, testing and benchmarking datasets of microscopy images by combining physics-based simulations and diffusion models. The codebase and datasets generated through this project will be made entirely open-source to fuel current and future AI-focused biomedical research.