Mainly size. weight,portability and hand holdability.
I have a 300mm f4.5. Length is only 160mm. Weight 880g
Compare to a monster of a lens if it is 300mm f2.8 at 247mm and 2495g.
The former can be part of the regular kit in the camera bag.
The latter will be more for specialized/planned outings since its so heavy/large and will often need a tripod.