you can stand your grounds, guards are mostly not armed and even if they are, they woun't use it in cases like this, but they can chase you out of the property ( you're a visitor/guest and they are the enfocer of the property and thus have the final say ), if you refuse they will hold you and call the police than charge you for transpassing ( ask you to leave and you refuse so you're transpassing ), If you kick up a fuss and cause a commotion, that gives them even more reason to call the police.
If you're on public property:
Ther guards can call the police if the building is a gov building / MRT station / Stat Board / etc. If the guards are the police themselves, they can cross the road and question you on the spot. Once I stop across the road from a police station and was making a phone call in my car. The police cross the road and ask what I was doing. I told him that I was making a call and didn't know that I can't stop there. He just told me that it is OK to stop there but they just have to be careful of people stoping near police stations and need to check what they're doing.
Still think you don't have to answer to anyone esp. if you in a public place? Well if you don't mind the inconvience or taking a risk, go right ahead.