Knuth's Law: Premature optimization is the root of all evil.
But the full quote is more nuanced: "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%."
The wisdom isn't "never optimize" but "optimize at the right time with the right data."
Optimize when you have performance problems, data showing bottlenecks, and understand the cost/benefit trade-offs.