Teflon, which is one manufacturer's name for polytetrafluoethylene (PTFE), is not poisonous. This chemical is inert (not active or reactive), so if it's ingested, it will travel through your system without being absorbed. So, you needn't worry if you accidentally eat a bit of it (though you probably wouldn't want to make a meal out of it).
There is some danger involved, however, when heating pans treated with PTFE to very high temperatures. When heated to over 300 degrees Celsius / 572 degrees Fahrenheit (a temperature you won't likely reach intentionally on your stove or in your oven), PTFE releases fumes. These fumes can cause an illness resembling the flu, with symptoms including tightening of the chest, mild coughing, nausea, and sweats. It's called polymer fume fever, and it's rare.
Birds, however, are a different story. You'll find lots of testimonials on-line from bird owners that heating PTFE products, even to normal cooking temperatures, releases sufficient fumes to kill their birds. So you might want to be careful with non-stick pans coated with PTFE if you've got birds in the house.
You can prevent a lot of PTFE flaking by not using sharp, metal utensils; they scratch the non-stick surface and cause all that flaking. Instead, use wooden or plastic ones. The scratches and flakes won't release dangerous poison, but they are unsightly, can release bits of dark PTFE specks in your otherwise spotless white sauce, and, most importantly, reduce the non-stick effectiveness of the pan. If your PTFE pans are getting flaky, it might be time to buy some new ones. If you're really concerned, you may want to switch to using cast iron, stainless steel, or other pans instead.