Yeah, for candids and casual shooting, a point-and-shoot (P&S) system would probably be better due to the more rugged construction (fixed lens) and smaller builds (can chuck it into a bag side pocket or even your pant pockets!). Ask any DSLR user... who would endanger their baby like that?
And if you argue with carrying a DSLR with you on hand or via neck strap... very hard to get candid photos without raising suspicions then =)
DSLRs are gotten primarily for flexibility (all the lens and accessories!), if you aren't going to get more lenses or addons, then just grab a P&S... good enough.