1) They have to be short enough to complete in a "reasonable" amount of time.
2) They have to have enough resolution to distinguish between good, better, and best.
3) They have to be representative of the real life problems that the person / computer will be solving.
Most benchmarks only meet objective #1. Further, benchmarks that only meet objective #1 are easily "game-able".
Benchmarks which meet objectives #2 and #3 invariably don't meet #1, which makes them hard to use as a benchmark.
1) They have to be short enough to complete in a "reasonable" amount of time.
2) They have to have enough resolution to distinguish between good, better, and best.
3) They have to be representative of the real life problems that the person / computer will be solving.
Most benchmarks only meet objective #1. Further, benchmarks that only meet objective #1 are easily "game-able".
Benchmarks which meet objectives #2 and #3 invariably don't meet #1, which makes them hard to use as a benchmark.