Recent studies have suggested that language production abilities decline in patients with small vessel disease (SVD) a pathology that is one of the main contributors of cognitive impairment in older adults. The loss of microstructural integrity in multiple white-matter fiber pathways has been previously associated with cognitive problems in people with SVD. Our previous work has shown that the presence of white matter lesions (WML), the most common neuroimaging marker of SVD, at the intersection between the anterior thalamic radiations (ATR) and forceps minor (FM) was associated with lower performance in Category Fluency in patients with SVD. Notably, the presence of WML is just a small part of a bigger problem as damage to white matter fibers can occur in areas remote from the location of WML. Importantly, evidence is lacking on whether performance in the Category Fluency test is associated with lower microstructural integrity in the ATR and FM. In addition, the uncinate fasciculus (UF), a ventral white-matter tract relevant for language, overlaps anteriorly with the ATR and FM and is therefore of interest for the present investigation. Our aim was to assess the association between Category Fluency scores and measures of microstructural integrity in ATR, FM and UF in patients with SVD. Diffusion tensor imaging is a suitable technique to investigate changes in the integrity of white matter fibers. We obtained a measure of microstructural integrity by quantifying fractional anisotropy (FA) and mean diffusivity (MD) by using TRActs Constrained by UnderLying Anatomy (TRACULA). The total mean FA and MD were calculated for FM, left and right ATR and UF in 221 patients with SVD (101 women, mean age 70.1). Mean diffusivity in FM (p = 0.000), in left (p = 0.019) and right ATR (p = 0.004), and in left (p = 0.034) and right UF (p = 0.013) were associated with worse Category Fluency scores. FA in FM, ATR and UF was not statistically associated with Category Fluency scores (ps > 0.05). In line with previous studies, we find that MD is a more promising biomarker of microstructural integrity in SVD relative to FA. The present findings extend our previous results, confirming an important role for FM and ATR for Category Fluency abilities in patients with SVD, and provide evidence for a link between microstructural integrity in FM, ATR and UF and the language decline in SVD. Our results have potential clinical implications in revealing an overlooked deficit in SVD and in suggesting a critical role of the integrity of FM, ATR and UF in this population.