Why not give a second hand 70-200/2.8L Non-IS a shot? A really solid performer and it's been appearing on BnS rapidly these couple of weeks at around 1.5k. It's 70-200 season now, so prices are down and consequently it's a good time to buy one. IMO, it whoops the butt of the 70-200/4L, which I upgraded from.
To be fair, the colour and sharpness look the same, but the extra stop makes heck of a lot of difference shooting in low light and also the ability to shoot at f2.8 for thinner DOF is a real boon sometimes. Well worth the additional 500 you'd need to fork out IMO.
IS is good, under certain circumstances, like low light static concerts or when using long telephotos like a 500 or 600mm, but for the most part it's overrated and can be done without. I'm really slim built as well(166, 52kg) but I've no problems using it for an extended period or getting off sharp images at 1/80 at 200mm. Just work on your technique and save the money you pay for the IS. If you really can't take it, get a monopod. Poor man's IS.