I think there are several factors, but main thing is what you want to achieve?
1. Are you bringing other lenses eg bring wide zoom and telephoto zoom, and bring 50mm to cover the gap. Or bring one full range zoom, but prime with big aperture for low light.
2. As others have said, what is your style and preference. Some like angle of view/perspective of 35, others like angle/perspective of 50.
3. If you are planning to use prime to supplement other lenses e.g. zooms or other primes, then are you used to changing lenses on the move? Sometimes, when travel in group or with others and everyone rushing, either may not have time to select prime, change lenses etc. or paiseh to make everyone wait while you do that.
4. Also depends on environment. For those who sayang their eqpt, may not be so keen to change lenses in dusty environment or if raining. In crowded and not so safe places, changing lenses can attract unwanted attention.
5. Focal length of prime also depend on what you are likely to shoot. If you are going to shoot family, friends group in front of big monument, may need wider angle. Same for narrow or confined places. If there is more space, then you may not get enough isolation with wider prime. As others have said above, sometimes no space to go forward or backward to get better FOV.