The concept of brain age (BA) describes an integrative imaging marker of brain health, often suggested to reflect aging processes. However, the degree to which cross-sectional MRI features, including BA, reflect past, ongoing, and future brain changes across different tissue types from macro- to microstructure remains controversial. Here, we use multimodal imaging data of 39,325 UK Biobank participants, aged 44–82 years at baseline and 2,520 follow-ups within 1.12–6.90 years to examine BA changes and their relationship to anatomical brain changes. We find insufficient evidence to conclude that BA reflects the rate of brain aging. However, modality-specific differences in brain ages reflect the state of the brain, highlighting diffusion and multimodal MRI brain age as potentially useful cross-sectional markers.