Hi, I just been to PD last month. This is how I got there.
From Singapore Take SBS bus 170
http://www.sbstransit.com.sg/journeyplan/route_information.asp?qdirect=1&qservno=170&qpoint=NO%20LOOP&dispno=170&qstart=LARKIN%20TER&qend=QUEEN%20ST%20TER to JB Larkin Bus Terminal. You can buy a bus coach ticket from The Larkin Terminal to 'Seremban'. The coach should bring you till 'Seremban' town's bus terminal. At this terminal, take another local bus(something Like our SBS or SMART. There should be sign board indicate to 'Port Dickson'). Take it till the end. Sorry, that I couldn't remember the coach coy name and the Seramban Bus number too. But it shouldn't be difficult to find when you look around the terminal. This will take almost half a day by this method. Hope this help.