I think that size does matter
Oversized poodles (so called "royals") can have proportions out of balance and extra weight can put extra pressure on joints and cause arthritis in later life.

Also, I would guess that oversized poodle would have even deeper chest and be more prone to the bloat.
Reputable breeders do take great care to produce puppies of a certain size and size is always considered when choosing a complementary Stud for a Dam.
In CA it is illegal to sell kittens and puppies before they are at least 8 weeks old and I would guess there is a good reason for that.