What stays with you more: what was said, or what wasn't?

I've been editing longer than I like to admit, and lately the cuts that feel right are the ones where the door never opens. The pauses, the room tone, the thing a person almost says. Curious what it's like for the rest of you — do the silences hold more than the speech, or is that just me being too long in the cutting room?