Sounds scary... I switched from sling bag to backpack because of this reason too. Lopsided while shooting, really not easy.
That said, the load of the backpack still bears down on my shoulders, causing my neck to shoulder muscle to sore. Sooner or later I will get the waist system from think tank which I believe will help alleviate the weight substantially off my back!
Home is where the heart is.