Am i a horrible friend?

Personally I think it a very normal reaction to have. If he has these insecurity problems it's not your burden to carry all the time. If he has serious problems then maybe he should seek professional help. While friends are good to talk too they lack two things usually: Confidentiality and the ability to be objective. But to rephrase my answer it is normal for you to feel this way about your friend.