Approximately 16 weeks after conception, even before they have ears, an unborn baby develops their sense of hearing. Some of what it hears is white noise, sounds muffled through flesh and bodily fluids, but mostly it just hears a heartbeat and its mother's blood sending much needed nutrients and oxygen through hardworking veins. Forget about what they hear, the fact is that 20 weeks before they're born, five months before they are people, fetuses are listening.
You would assume sight is your most important sense, and for most everyday activities it probably is, but sound is much more vital to our survival (the caveman could hear the Saber-toothed tiger in the bushes before he could see him). . We can hear at 360 degrees spherical (that's every direction). Sight is only usable 180 degree at a time. We can close our eyes but our ears are always open. We hear all the time, we may filter it out, ignore it, forget about it instantly, but sound is always coming into our ears, and therefore, our brains.
v. heard, hearing, hears
1. To perceive (sound) by the ear
intr.v. listened, listening, listens
1. To make an effort to hear something.
Definitions from The free dictionary
We hear everything, but we listen selectively. Improvisers train ourselves to listen to everything (or should) because we understand how important it is. How every subtle intonation, every emphasis of every syllable changes the meaning of every sentence (and therefore every scene). We put the effort in to understand what was said. If all I do is hear you, I miss the most important part of what you're saying... I miss what you actually mean.
Because there is something that comes along with listening, with putting in that effort: interpretation. Soaking in the meaning of the words. Every word in the English language has multiple meanings, and when you listen you can tell the difference. If I interpret your meaning, when I respond, I'm responding to what you said and not just the words you used. It's the difference between a good improviser and a great improviser.
Words will always come between improvisers on stage. They'll come between people at jobs, and even lovers in beds. So put the effort in to really listen, and let the words come between you, not like a wall, but like a bridge.