Anxiety is a terrible thing. I went through a period of time after the birth of my 2nd daughter where every time I left my house I would have an anxiety attack. My daughters are 14 months apart so when I had a newborn (my 2nd daughter) ... my older daughter was only 1 year and 2 months old...still a baby. I would fear that I would forget one of them in the car, lose them in the store, not be able to get to a restroom in time, afraid to let someone else drive (then I wouldn't be in control), etc. All of my fears were irrational and I knew it...but I just couldn't control the anxiety. My rational mind knew it was silly to have anxiety about those things. My anxiety issues lasted a couple of years. I lost a ton of weight (wish I could do that now!), detached from my husband (we are now divorced), became angry and on edge all the time and probably was no fun to be around. I eventually got counseling and medication. I am still on medication and I think it keeps me grounded and even keeled.
I guess what I am trying to say. Get some help for your anxiety. It is no way to live for you or your wife. It is hard on you emotionally and physically. I have not had an anxiety attack in years and I never think twice about doing anything. There is life after anxiety.
Good luck to you.
Sisu