you may want to consider going to the temple earlier, to watch the people prepare. there will be some interesting sights, such as some of the ceremonies/rituals going on before they set off.
i can't remember the exact timings, but it's probably good to go quite late, maybe nearer to midnight. that's when the men carrying the big kavadis set out.
for those not used to this, it can be a rather intense experience. lots of crowds, sounds, colours...
if you're curious, don't hesitate, just go. but be respectful, it's a big day for the kavadi bearers, and the physical and mental pressure is tremendous.