you really need to define what is affordable.
for portraits, canon's definitive portrait lens is the monster 85mm F/1.2L. but that will set you back $3k+... alternatively there is also a 85mm F/1.8 which is under $1k.
similarly, for sports, you can opt for an Image stabilised 70-200mm 2.8 constant aperture zoom, for close to $3k...
there are also cheaper zooms , but without IS and with higher aperture ranges..
it is definitely worth spending money on good glass. the reason is because lenses will outlast the lifespan of your bodies.
also, whilst you will lose quite a lot when you sell your DSLR 2nd hand in a year or 2, lens prices are very stable...
if you look over at buy and sell forums, you will see many of the high quality or better quality lenses going for only 10-20% off their sale price, and these can be a few years old.
even the crappy canon 50mm F/1.8, which retails for about $130, fetches $100 in the 2nd hand market.