Test2::V0 docs should contain version numbers of features added later

[bbrtj]

Test2::V0 is promised to never break backward compatibility, but instead create a new bundle with incremented version number - that's awesome. However, it is not constant, and some keywords were added later, as I found out today.

A test file using check_isa has been failing today in my CI for me on windows / perl5.30. As it turned out, this keyword was added in version 0.000139, back when it was a part of Test2::Suite. However, finding this version number was not easy because:

• Test::Simple changelog doesn't contain changelog of Test2::Suite
• Test2::Suite in its old form can no longer be found on metacpan
• Test2::V0 documentation has no mention of check_isa being added later

To find this version number, I had to:

• find an archived repository for Test2::Suite on github
• find "ISA" in its changelog (no results for check_isa), which pointed to a version number
• find a merge request in git history and confirm that it added check_isa

This is hard and it requires access to original Test2::Suite repository (cannot be done through metacpan alone). Instead, my suggestion is to add a list of features/keywords along with version numbers they first appeared in somewhere in the docs of Test2::V0, for all features/keywords which were not a part of the original release. This way it will be easier to know what minimum Test2::V0 version to put in dependencies.

[exodist]

First off, I agree, it would be good to add the "added in version xxx" to all exports in Test2::V0, as well as Test2::Tools::*

Second, the Changelog does contain everything from Test-Simple, Test2-Suite, Test-Stream, as well as just Test2.
https://github.com/Test-More/test-more/blob/master/Changes#L1050
https://github.com/Test-More/test-more/blob/master/Changes#L2164
https://github.com/Test-More/test-more/blob/master/Changes#L2394
https://github.com/Test-More/test-more/blob/master/Changes#L2587

They are just several different logs before the merge, one combined log after it. So not the most chronologically useful, and also unlikely to have everything you were looking for.

Also, I think I need to refine and write out a proper document covering how api breakage/changes work. But essentially this:

• If at all possible, do not break existing API's
• Avoid removing functions/exports/arguments
• Avoid making things that currently work resulting in exceptions
• Avoid making things that currently pass into failures, unless there is a demonstrable bug that something should never have passed
• When making such changes avoid breaking unmonitored test suites. It is ok to throw an exception or fail a test if AUTHOR_TESTING is set, but without that env var the failure should be marked TODO automatically, and diags are better than exceptions.
• It is ok to add new functions/arguments any time, unless the function is likely to conflict by being a common word or part of another testing tools typical exports.
• But always add a note for what version it was added in.

I want to refine this more, and am open to feedback on it. My goal would be to handle this at, or before the next perl toolchain summit.

[bbrtj]

Oh okay, then I did not look far enough into the changelog and finding check_isa returned nothing since it had no occurences in original Test2::Suite changelog either. Changelog alone would not be enough to find out the version for sure anyway, as Merge PR to add ISA checks is not descriptive enough, at least to me.

I am a bit undecided when it comes to Test2::V# bundles. In perfect world they would be absolutely constant, but that would mean that bugs in the underlying tools would have to be kept indefinetly. Adding completely new keywords to them seems okay-ish, but adding keywords which already exist but were not included originally seems wrong. Not saying it ever happened, since I don't know that. My opinion would be that even though the underlying tools may change behavior, the set of keywords/features exported by Test2::V# bundles should neither grow nor shrink. If tools change, changelog should absolutely include the name of the tool (so it is searchable) and good enough description so that it is enough to understand the change at large. Bundle's documentation should also track the changes in the underlying tools, just for clarity - should not be too much work.

[exodist]

Please take a look at #1052
It was partially influenced by your request here.

[bbrtj]

I'm a bit undecided about it. I don't personally like the completely new approach with T2 handle that much, it seems like a major change of course compared to how testing in perl has been done since forever. But then it gives an option to import symbols as it was before, and even through T2 namespace. This seems like too many usage variants, which will be confusing compared to how straightforward V0 is. But hey, if T2 handle fixes problems then I could live with that.

What I really like about T2 handle - I could pass it to some test libs and have them use it instead of importing the library once again in that file. That's great.

If it comes to new symbols being added later, it still gives itself room for that, which means you still need to depend on a versioned bundle with a minimum version specified. So it still seems a bit contradictory to the idea of versioned bundles in my eyes.