I think should not be a problem if you are not using flash... And also, do not take when the golfer is about to take the shot if you are using DSLR as the shutter sound might distract them (unless of course you are using 600mm lens). I have seen on TV when a golfer snatch a journalist's camera away when he distracted her from her T-shot.....
