what i prefer is to always have the o-ring off, unless i'm going diving.
if you leave it on, its going to get stretched and takes on a fixed shape = doesn't seal as well = leak = OMFG! = ruined dive trip = sian...
also more likely to accumulate dirt, grime, hair, dust, sand when on the housing than in a ziploc.
also if you leave it on, seal the housing, and leave it in the sun, its likely that you will damage the housing.
YMMV!
still...i've seen divers who chuck the housing, o-ring and all into their dive bag with BP and regs which get stacked on by 20 other dive bags, and have no problem at all.