While neuroimaging research on language production has traditionally focused primarily on grey matter, several recent studies highlight the involvement of ventral and dorsal white matter pathways. A debated issue concerns the exact functional role of these pathways. The ventral pathway has been suggested to underlie top-down control in language production, but the functional roles of each specific white matter tract within this pathway, like the inferior fronto-occipital fasciculus and uncinate fasciculus, have not yet been elucidated. To investigate the involvement of the inferior fronto-occipital and uncinate fasciculus in top-down control, 15 patients with primary progressive aphasia (PPA), an acquired language deficit due to neurodegenerative disease, and 22 age-matched healthy controls performed a picture-word inference (PWI) task including an incongruent (semantically related word, e.g., word cow on the picture of a horse) and a neutral (e.g., XXX on the picture of a horse) condition. The stimuli were made of 12 high frequency words from two semantic categories (animals and fruits). The difference in reaction time and accuracy between semantically related and neutral picture-word pairs served as a behavioral measure of the participants’ top-down interference control. Furthermore, the microstructural integrity of the inferior fronto-occipital and uncinate fasciculus was calculated as a neuroanatomical measure with diffusion tensor MRI and tractography, expressed in Fractional Anisotropy (FA) and Mean Diffusivity (MD) values. A linear mixed-effects model revealed that patients were more susceptible to the PWI effect in reaction time compared to controls. Moreover, the integrity of all three tracts was altered in patients compared to controls. Importantly, the integrity of the IFOF was not associated with the PWI effect in reaction times or accuracy. The integrity of the ILF was associated with the PWI effect in reaction times, but not accuracy. The integrity of the UF was associated with the PWI effect in both reaction times and accuracy. These results indicate that PPA patients manifest impaired top-down control processes in language production and that these processes are mediated by the microstructural properties of the inferior fronto-occipital and uncinate fasciculus in both PPA and healthy individuals.