The f number is derived from a ratio of focal length to effective aperture diameter.
Indirectly, since a faster f stop needs a larger effective aperture diameter for a given focal length, u will physically need a larger lens construct.
No hard and fast rule. There are many other factors such as lens design etc ..
But if you have a 50mm f1.8, 50mm f1.4, and a 50mmf1.2,
The faster lens 50mmf1.2 will be the bulkier / larger diameter lens.