I shoot primarily with primes (17, 24, 35, 50, 100, 135)
Using primes will certainly add weight to your bag, the best primes tend to be rather heavy. What you do gain is significantly better colour and contrast.
If you want to go for prime lenses, I would suggest the 35/1.4L, you wont look...