How much test fixtures is needed per language?

[PizieDust]

Hi @mickeynp

@tmcgilchrist and I are working on adding support for the ocaml language to combobulate. We noticed we may end up with too many test fixtures if we wanted to properly test navigating the ocaml syntax: so here are some questions:

• What is the typical test coverage you expect for any language?
• Is it necessary to always have one file per tested feature?
• What do you think about the balance between test coverage and test suite size and do you want to have a limit on the test suite size?

For context, we plan to write over 25 test fixtures

[mickeynp]

Hi there!

Good question. I don't maintain test coverage in any of my projects because I find that aiming for a magic number like 100% is often counter-productive in terms of manhours vs reward, and it's not like it completely eliminates bugs.

The reason you want a reasonably thorough set of fixtures and tests is mostly if you edit the actual combobulate code: it is very easy indeed to tweak something here and have it only fail all the way over there.

The go test fixtures use large expansive example cases because I realised that is usually good enough instead of lots of small isolated tests. After all: you're just wasting a few bytes of storage. If you can re-use the same file content for multiple tests? Go for it.

It sounds like ocaml has a lot of weird edge case language constructs if you need that many? I mean, if that is what the language needs, it's fine. Ultimately the real goal is to ensure that particularly sibling navigation works correctly as it is used in many different parts of combobulate.

[PizieDust]

Thank you. This is very helpful

[mickeynp]

By all means ask questions if you have them. I'm happy to help.