Find the word definition

Wiktionary
penalty function

n. (context mathematics English) Any function that applies constraints to a maximum or minimum problem