Yes normally just bit of dust won't affect. Else those wearing spec will always be seeing dust in front of their eyes. We will also find some scratch line on our spec but won't be seeing it unless it really deep one.
I had dust inside my previous lens b4. It doesn't show up unless I down the aperture all way (eg f16, 22) and really go "find" the dust on taken photos.
Dust and smudges on specs are clearly visible and get annoying much faster, especially when the light is coming from front (eg. lamps, low sun). The reason is the distance to the eye. Similarly: the speck on front element is never visible, but specs on rear elements or sensor (correctly: filter before sensor) will show up when using small aperture.