English proverbs, idioms and expressions related to this theme.
Don’t throw the baby with the bathwater.
suggests that