Wiktionary
throw the baby out with the bathwater
vb. (context idiomatic English) To discard, especially inadvertently, something valuable while in the process of removing or rejecting something unwanted.