I think it’s because they remind us of the purity of Christ.
Nothing like a precious newborn infant to remind us of the pure love of God. This new life is totally dependent upon its parents for EVERYTHING just like a spiritual newborn is totally dependent upon Christ for salvation and sanctification!
-Mark (little known fact about me – I love babies! I could work in the nursery every Sunday – well, maybe every other Sunday!)