Some kind of "conformance" suite for in-step tests?

[thejcannon]

There's a definite pattern for the core tests a tool should have:

• "Just checkers":
  • check exits nonzero on bad file
  • check exits zero on good file
  • check exists nonzero on both
• "Just Fixers" (i.e. a formatter):
  • ...(Same as "Just checkers")
  • check_list_files exits nonzero with filename on bad file
  • check_list_files exits zero with no output on good file
  • check_list_files exits nonzero with just bad filename on both
  • check_diff exits nonzero on bad file with the diff
    • (ideally, a way to expect that hk parsed the diff)
  • check_diff exits zero on good file
  • check_diff exits nonzero on both, with bad file diff
  • fix exits zero on bad file, and fixes the file
  • fix exits zero on good file
  • fix exits zero on both (and fixes the bad file)
• Combos (checker/fixers)
  • ...(Same as "Just fixers")
  • fix exits nonzero when it fixes some but not all issues

Ideally, steps can use/configure/opt-in to a pre-fabricated set of test cases for the above.

Something like this would be helpful to easily find (and fix) bugs like the pkl format one in #489.

Additionally, it would also help strengthen the case of suggesting conformity a la this pkl format issue or this shfmt issue

[thejcannon]

(Ideally it'd also allow us to flex #462)

[thejcannon]

One way to test check_list_files and check_diff that isn't shit is to add tera to StepTestExpect.

We'd want to unify (more) the code in test_runner.rs and step.rs (as far as tera context goes).

[jdx]

in general we should use tera on everything we think we might want tera in the future even if we don't have a current use-case since adding it could be a breaking change

[thejcannon]

Oh so specifically what I meant was (and maybe this is what you're saying) that we should add a tera key to the expect where users can expect the values of tera variables (e.g. for the check_list_files case, you'd expect the files variable to be a subset of the input files). Same for making sure the workspace indicator is working.