Yes, I believe that 4 stops would be a great improvement, allowing me to increase my shutter speed and probably drop the ISO
You will be dealing with very nerrow depth of field btw. And less flexibility on your framing when shooting on prime lens too.