It's not absurd, it's just a symptom. A really frustrating one. I get this too. I find that scheduling a time for it can help - like, I'm going to clear my voicemails tomorrow at 5pm, or something like. I also find that putting them on speakerphone helps, for some reason. I can keep some distance or something.
My email stress lowered when I started using the preview pane - so I can already see the first few lines of the email. Also, my email is configured to put anything from family that I'll likely not want to see right away into a separate folder.
I hope some of this helps.