Wish: AssertNotImplemented() should return True #13
Description
AssertNotImplemented() should return True, not False. Here's why:
Reason 1 (tautological): If a test is not implemented, then by definition it's always true that it is not implemented.
Reason 2 (practical): By returning False, AssertNotImplemented() trains us to ignore failing tests.
The general rule of unit testing is that all test results must be green before code is committed or shipped ("all green or no go").
When tests that are not implemented return False, we train ourselves to ignore those red results because we know they're expected. This is a problem because if we get used to seeing and ignoring any red results, even if only those from tests that are not implemented, we might miss a red result from a newly failing test that is implemented.
This can be particularly problematic in larger test suites.
Example: A test suite has 100 tests. The 76 that are implemented return True (green). The 24 that are not implemented are there as placeholders for future tests, and they return False (red). We become accustomed to seeing and ignoring the 24 red rows every time we run the whole test suite. If we then modify or refactor a method and its test subsequently fails, we might not notice there are now 25 failing tests instead of only 24. We might go ahead and commit or ship the code even though not all test results are green, and if we do that we have just released a bug.
On the other hand, if tests that are not implemented return True (green), we know all red results require action and we know we can't commit and ship the code until all tests are green.