Description
I think we should come up with a more sophisticated model for test execution order.
Two goals I would want to achieve by changing the current naïve logic:
Have failing tests fail earlier
For this we would need to keep some stats about the most recent test suite execution.
Based on this we would then partition the sub-blocks of each context into two groups:
- Tests that failed
- Tests that succeeded
The runner would sort the sub-blocks by moving the group of failing tests to the front.
Prevent tests from depending on execution order.
Within these groups then we would want to randomize the order of sub-blocks to prevent accidental dependencies on execution order.
In principle rspec does already prevent one from depending on execution order through its sophisticated environment isolation. In the case of shared mutable state (sharing a database connection behind a mutex between contexts e.g.) however rspec cannot give such guarantees.