Background: With a growing, younger population of head and neck cancer survivors, attention to long-term side-effects of prior, often radiotherapeutic, treatment is warranted. Therefore, we studied the long-term cognitive effects in young adult patients irradiated for head and neck neoplasms (HNN). Methods: Young to middle-aged adults with HNN (aged 18-40 years) and treated with unilateral neck irradiation ≥ 5 years before inclusion underwent cardiovascular risk and neuropsychological assessments and answered validated questionnaires regarding subjective cognitive complaints, fatigue, depression, quality of life, and cancer-specific distress. Additionally, magnetic resonance imaging (MRI) of the brain was performed to assess white matter hyperintensities (WMH), infarctions, and atrophy. Results: Twenty-nine patients (aged 24–61, 13 men) median 9.2 [7.3–12.9] years post-treatment were included. HNN patients performed worse in episodic memory (Z-score = -1.16 [-1.58–0.34], p < 0.001) and reported more fatigue symptoms (Z-score = 1.75 [1.21–2.00], p < 0.001) compared to normative data. Furthermore, patients had a high level of fear of tumor recurrence (13 patients [44.8%]) and a heightened speech handicap index (13 patients [44.8%]). Only a small number of neurovascular lesions were found (3 infarctions in 2 patients and 0.11 [0.00–0.40] mL WMH), unrelated to the irradiated side. Cognitive impairment was not associated with WMH, brain atrophy, fatigue, or subjective speech problems. Conclusions: HNN patients showed impairments in episodic memory and an increased level of fatigue ≥ 5 years after radiotherapy compared to normative data. Cognitive impairments could not be explained by WMH or brain atrophy on brain MRI or psychological factors.