So if I do agree is it up to me to say each time that I'm struggling or should he be aware enough now to know that I probably am?
I don't know him, and that matters. But, yes, there's a good chance you probably do have to say something. Because it might be harder for him to figure out what's going on with you than you think, especially when he's distracted, so to speak. Maybe some kind of safe word? ("No" would work.)
I think it's really easy to over estimate how easy it is for people to understand what we're telling them. Here's an example. I have a good friend that I've known for years. She's been taking medication for anxiety since she was in high school. She's a really nice person with a lot of empathy. She is one of the few people who knows I have PTSD and am seeing a therapist. One day, we'd met up for lunch. She was asking how things were going. I told her a story about a lady I'd had to work around one day who had me totally freaked out because she's a tiny little woman, apparently quite timid (in other words she reminded me of my mother) and she kept trying to get my attention by sneaking up on me and touching my arm. Usually coming from a direction where it was hard to see her coming. I said it was hard to keep from swinging at her, just as a reflex. My friend said, "I know JUST what you mean. I hate it when strangers touch me too." I paused. I thought to myself, "Maybe you know what I mean. I kind of hope you don't actually know how I'm feeling........" Then she, joking around, reached across the table and touched my arm. AFTER I pealed myself off the ceiling, she, with a totally shocked expression said, "Maybe I don't know what you mean."
The thing is, we take our view of the world for granted. Because it's our view of the world. Other people do that too. And, they may be super well intended and TRY to understand, but a lot of times I really believe they only THINK they understand. And we only think they understand too. So, if he's a basically nice guy? He might THINK he gets what a big deal this is for you, but I'll bet he doesn't. And I'm not sure how you get it across to him.
I really think you're approaching all this with a great attitude. I'd keep that up, if you can. Look for answers and new information, not a chance to assign blame, just like you're doing.