Easier said than done I know, but stop feeling guilty for caring about more than just him. It won't do you, him, or your daughter any favours.
Take comfort from the fact that he obviously loves, needs and misses you (as he should!), but that he understands and accepts you need to be doing...