It only matters as much as the kind of lens you put in front of the camera. Using a cheap filter is akin to using a cheap lens. It doesn't matter how expensive your lens is, and how good it is, if you have the bottom of a coke bottle as your filter, you'll take rubbish pictures. (Unless you call it fine art :bsmilie: )
That said, you are right in that it largely doesn't matter. "Lousy" lenses frequently turn out more than good enough performance to satisfy the majority of photographers if they judged it solely on its optical quality and not just it's brand. The same would apply to filters.