Just a suggestion that might work.....so far, it seems to work.
I did have people coming up telling me off when I took street shots in wet markets and streets (of course, they're nice people around that actually waved and posed). Anyway, those wet blankets are usually the stall owners (if I stayed there long enough) and shoppers (I still remember I was shooting the ceiling of the market towards the direction of an middle-aged HK couple and the guy ran over and said,"U can't take pics here!" He ran off promptly after that before I could register anything. Weird.) Back to the suggestion....When I got back to my bodybuilding regime, and I now look intimidating again... People generally just leave me or probably just stares. So i guess...looking like a bad arse might actually stop all these unpleasant encounters..