Doesn't matter for so much for hobbyists, but does for pros and heavy users. Consumer grade lenses start to feel wobbly and optical alignment may be affect, especially those plastic lenses with extending barrels. Aperture blades can get stuck when dust accumulates internally. Tiny screws can drop out over time.
The next accidental knock could be that lens' last. And it could happen in your once-in-a-lifetime $8000 Europe tour.
Are all consumer lenses that bad? Plastic is light and durable right? Do not get me wrong. Not telling that they are better than the nikkors/canons. But are they really that worse?
BTW is anyone really that careless with a lens worth $$$? Pros should maintain stuff better than amatures right? Not underestimating what an unfortunate drop can do, in which case build definately matter tons!