The 40D is your best bet - best feel and fastest performance by quite a margin. The 450D is feature-rich but if you have big mits, you won't feel very comfortable holding it for long, at least in my experience.
Why not get a feel of both cams and see which one suits you better? It's not so much about what the cam can do (since the D40 and D80 are all reliable), and images really depends on your own style. So get one that feels more comfortable and look to expand your gear!