Yeah, I know what you mean. But to me it's much easier and faster to adjust shutter speed w/ the rear dial instead of the speed dial on the top. But I just don't want to dig into menus to change shooting modes.
Besides, most users like me would shoot in Aperture Priority (Av) mode anyway. We are gonna set the speed dial to A, and won't touch it anymore. It's like waste of space, better use it for shooting mode dial.
Since getting the X-E1 more than 2 months ago, not once have I ever used the speed dial. I always shoot in Aperture Priority mode.
Only time I ever use shutter priority is if I want to do panning, but I don't fancy doing panning with the X-E1's unpredictable AI-Servo (or equivalent). No doubt the speed dial adds to the retro styling, but perhaps something more useful could have been put in is place.