It's not bad to say no... and when those of us who, as children, were not allowed to say no, get in a situation like yours, we tend to want to make things right for everyone but ourselves. We are not obligated to make the world, or anyone happy, we are here to gain victory over the past and start anew, even if that takes getting past mistakes. You made a decision not to go; you knew what might happen and made the choice to avoid it. Don't second guess yourself; go with your gut instinct.