On a macro-level, how would you even begin to write tests for a search engine or some stock market bot or other notoriously hard problem?
These problems are hard because you evaluate your results with some kind of "quality score" instead of a simple "pass/fail" metric. You need to adapt your testing strategy accordingly.
One useful strategy is to define a decent metric, and try to maximize it. Let's say that you have two sources of data: (1) A list of pages that should rank highly for specific queries, and (2) a list of links that users clicked on and stayed at without coming back to click on other links < 30 seconds later.
Your goal is to write a search engine which ranks these links, on average, as highly as possible for the relevant queries. You do this with the usual techniques of experimental science: Split your data in half, develop against part of it, and hold a part in reserve for your final tests.
If you make a clever change, and your average "good link" position drops from 1.7 to 3.9, you back your change out and try again.
One of my clients actually did something similar with mapping software. Before deploying new code (or a new version of the map data), they ran extensive automated tests, and flagged anything weird for human review. I wrote them several test harnesses, one of which discovered a situation where the driving directions took a right hand turn off an overpass and tried to merge into traffic below... :-)
Granted, these types of "tests" are essentially very high-level integration tests. But they serve the major purpose of tests: Automatically ferreting out disastrous mistakes before you ship to customers.
I think this is another misunderstanding because I worded that question so poorly. Apologies.
Yes splitting your data into a training set and a test set is a really good idea for problems like these.
But I'd still argue it's near impossible to generate decent grading first, TDD-style, with only a superficial knowledge of the problem you're trying to solve in that case.
Say you've got one obscure Star Wars character in your test set. Top results are probably imdb and wikipidia and that looks good. Then you've got a slightly less obscure character in the test set. And you stick with imdb and wikipedia as the top results for testing purposes. But he's got a huge page on wookiepedia. He's got a huge fansite. He's got a huge personal page. A twitter account more popular than Austin Kutcher. All popular enough to bump IMDB and Wikipedia out of the top five. Now the test you wrote is broken, and you're coding to broken results.
In that case I don't think we're capable of generating good tests to define the result set FIRST, TDD-style. I don't think we're capable of specifying the application behavior FIRST, BDD-style. I think its better to play around with the data and algorithms first, until you start to get a gut feel for the data. And then you can write some decent tests for the test data.
Same with mapping software. It's probably a good idea for the shipping product to have a test suite that does something like generate 100 random or not-so-random trips, then make sure that: you can get there from here, You can do it in +/- 5% of the time/miles that we've already esablished, etc. But I question how much value those tests have for day 1 or week one or even month one when you're writing the software. There's a lot of legwork before those will even come close to passing. And because you now don't have an un-broken build, people will potentially start ignoring problems with tests that should be passing week one.
And I know you're not saying this, but every time people argue about TDD, the TDD proponents seem to think that no-one else tests. Which isn't true at all. You do need tests. The question is when and how. And the answer isn't always before anything else.
These problems are hard because you evaluate your results with some kind of "quality score" instead of a simple "pass/fail" metric. You need to adapt your testing strategy accordingly.
One useful strategy is to define a decent metric, and try to maximize it. Let's say that you have two sources of data: (1) A list of pages that should rank highly for specific queries, and (2) a list of links that users clicked on and stayed at without coming back to click on other links < 30 seconds later.
Your goal is to write a search engine which ranks these links, on average, as highly as possible for the relevant queries. You do this with the usual techniques of experimental science: Split your data in half, develop against part of it, and hold a part in reserve for your final tests.
If you make a clever change, and your average "good link" position drops from 1.7 to 3.9, you back your change out and try again.
One of my clients actually did something similar with mapping software. Before deploying new code (or a new version of the map data), they ran extensive automated tests, and flagged anything weird for human review. I wrote them several test harnesses, one of which discovered a situation where the driving directions took a right hand turn off an overpass and tried to merge into traffic below... :-)
Granted, these types of "tests" are essentially very high-level integration tests. But they serve the major purpose of tests: Automatically ferreting out disastrous mistakes before you ship to customers.