How old are you? Once you are nearing 40s, long sightedness sets in and you need the diopter to compensate. I have a colleague whose sister, in her early 30s, suffering from long sightedness.
Do I need to worry? I find a -1 diopter more comfortable than a 0 setting even with glasses on so I've set up all my cameras this way. Certainly not anywhere near my 40's. Images are turning out fine though