When I go hiking, it tends to be with other people, a couple of friends or family. If I walk alone I find that I get anxious, jumping at shadows rather than any real danger. I think this is why I want a dog! But I think that as others have said, as long as you take precautions, you are as safe as you are in any other situation. It's about balancing the protection against the risk. I agree that the further out you go, the less people you see and for me that helps clear my head, although I do then panic at the people I do see because I question their motives, why they are out in the middle of nowhere. Although they are probably thinking the same about me.
I go out into the woods on my bike alone quite happily. Working on the basis that someone on foot can't catch me. In that situation, it is the worry of crashing and not being able to get help that stops me straying too far off the beaten path. But taking clothing/ a bivvy bag, food and water and tools for simple mechanicals is the sensible way of protecting myself from harm as much as possible.