I think something is lost in translation. Once upon a time, premature optimization was writing a subroutine in assembly because you "knew it would be slow." Now, people are using ill-fitted data structures and optimizing for developer time in the name of avoiding premature optimization.
These days, with the test suites we have and the highly decoupled architectures we have created, we can profile nearly instantaneously. Put ten million records (or more, depending on your expected volume) in a database and test your code against it. If your test begs for mercy, it isn't premature optimization to fix it.
From a startup's perspective, I understand the idea of failing fast, but getting some buzz and seeing your servers melt down is a good way to fail fast as well.
This was posted on HN recently. Skip to slide 18. In it, the guy talked about premature optimization, curiously, of the same thing: linear iteration over an array vs using a hash implementation.
His example was that in his youth, he was on the Doom forums, and he was doing mods, and took a look at the code that loaded the different asset files. And he was abhorred that the code used an array to linearly iterate over to find the asset to load. He started complaining on the newsgroups, and then Doom guys said you don't know what you're talking about, etc.
In this case, it was where that code just didn't matter in terms of run time, because the order of magnitude difference in looking up an asset and actually loading the asset was huge, and it just didn't matter. And because they used a dumb implementation, they could optimize for developer productivity and go on to code that mattered, like the rendering.
That said, I think the thing to keep in mind is, which parts of your problem space is of the core importance, and keep an eye on difference of order of magnitude in the different solutions you were thinking about implementing
I listened to his whole talk, and it was quite eye-opening.
And sure enough, just the other night, I ran into a really hard to find memory leak in some new code. At that point, I regretted tailoring a couple of strictly speaking, unecessary object as helpers, and just replaced them with arrays. It was then somewhat easier to nail down my memory leak.
Point being, that as soon as you need complete comprehension of some code, whether for performance tuning, or debugging, any reduction in the complexity of the code can pay massive dividends.
The easy, general lesson is you need optimize "in the large" from day one, meaning having architecture that won't bottle necks but avoid optimization "in the small", because the small can be changed easily.
On the other hand, I'd say that the test suite you use for optimization should be taken as much from your real world experiences or expectations as possible. It would be different from a suite of unit tests or even integration tests.
I'd suggest that your happy path integration tests would be a good place to start, coupled with a database loaded more heavily than you expect in the real world.
These days, with the test suites we have and the highly decoupled architectures we have created, we can profile nearly instantaneously. Put ten million records (or more, depending on your expected volume) in a database and test your code against it. If your test begs for mercy, it isn't premature optimization to fix it.
From a startup's perspective, I understand the idea of failing fast, but getting some buzz and seeing your servers melt down is a good way to fail fast as well.