This is how I usually do, and when I use this method, most taxi drivers I encountered (so far) won't argue with me because I guess I am correct, in a sense. I guess drivers are also worried that passengers may complain to the taxi company.
Usually I take note from that point onwards that he tells me that he is lost in which, either you will know or the driver will say something like he missed the exit/U-turn/whatever. I will use my handphone and take note of the pricing at the particular point. Until the driver gets back on the right "track", I will take note of the new rate on the meter. So if the difference between the first record and second record is $3 (for example), upon reaching my destination I will tell the driver that he has to deduct those $3 from the total amount.
So far, this method works for me. But if he turns hostile, just remember to ask for a receipt because the receipt will have all the information such as the license plate number, the time you board the taxi etc. At least you will have enough proof if there is a need to complain to the taxi company.
Just my two cents worth.
{mich}