Extracting meaning from a rapid and complex speech signal begins as a challenging task, exacerbated by the declines in hearing ability and cognitive function that frequently occur in normal aging. How does the brain successfully decode speech under conditions of auditory and cognitive challenge? Does added auditory challenge influence our ability to process linguistic information? I will discuss behavioral and neuroimaging evidence regarding the neural substrates of these processes, framing them in terms of a brain-based model of speech comprehension. Evidence from multiple sources is consistent with a shared resource framework of speech comprehension in which domain-general cognitive processes supported by discrete regions of frontal cortex are required for both auditory and linguistic processing. The specific areas of neural activity depend on the difficulty of the speech being heard and the hearing and cognitive ability of the listeners.