Simply put, only do it if you have enough confidence. If you are jittery then the above scenarios that you describe will come to reality more easily because of the psychological barrier.
Be gentle, and nothing will go wrong. The dude at Orient Photo whipped out some cotton buds and started cleaning my mirror when I told him the focussing screen was dirty, so I guess there's no real issue.
Another point would be that you might want to consider using a blower first, it is generally the safest thing to use on your body. And if it ain't broken don't fix it. If the dust on the mirror is not causing much problems, leave it alone.
Same goes for other stuff like dust on your sensor. If it's one small speck that is only clearly visible at apertures smaller than f/22.. Leave it alone, hehe!