According to the competition account of lexical selection in word production, conceptually driven word retrieval involves the activation of a set of candidate words in left temporal cortex and competitive selection of the intended word from this set, regulated by frontal cortical mechanisms. However, the relative contribution of these brain regions to competitive lexical selection is uncertain. In the present study, five patients with left prefrontal cortex lesions (overlapping in ventral and dorsal lateral cortex), eight patients with left lateral temporal cortex lesions (overlapping in middle temporal gyrus), and 13 matched controls performed a picture-word interference task. Distractor words were semantically related or unrelated to the picture, or the name of the picture (congruent condition). Semantic interference (related vs. unrelated), tapping into competitive lexical selection, was examined. An overall semantic interference effect was observed for the control and left-temporal groups separately. The left-frontal patients did not show a reliable semantic interference effect as a group. The left-temporal patients had increased semantic interference in the error rates relative to controls. Error distribution analyses indicated that these patients had more hesitant responses for the related than for the unrelated condition. We propose that left middle temporal lesions affect the lexical activation component, making lexical selection more susceptible to errors.