proof-of-concept: multi-app backend integration tests

[adghayes]

We've long talked about the possibility of building multi-app integration tests. I wanted to see if there was a low-ish lift way to get there, and landed on only doing backend-only "integration" tests.

We could spin up the frontend and backend for all apps. Playwright likely has the configurability to do this, provided we update some of our run scripts to be more flexible. It would probably be slow however and, if not carefully managed, flaky. If we could reduce the flakiness and didn't mind the slowness, however, it would allow us to cover more surface area than this.

The backend-integration-testing library here recreates the testing infrastructure for each of the backends and allows running them together. The test interface is the backend API rather than the frontend UI. This means that we are unable to test frontend-specific multi-app behavior. But the backend-only approach is a lot faster - I'm able to configure VxAdmin, vote at VxMark, scan the ballot at VxScan, and load the CVR at VxAdmin again in less than 7 seconds.

It's an open question of whether writing these tests would even be worth it. For many of the main integration points - e.g. scanning ballots - we already have a sort of integration test with ballot generation and interpretation that simply cuts out the hardware mocks and is far more efficient than what I made here. The compatibility of election packages and cast vote records is not explicitly tested, but is generally enforced by typing. I'm not in the weeds enough to know if there are bugs that we could catch with this sort of testing that we are currently unable to test, or if it could meaningfully speed up QA.

My favorite part of the testing API:

• able to persist files on mock USB drives between apps, signatures and file structure all managed and implicitly tested
• able to persist state of an app within a test instance. e.g. start at VxAdmin, go to other components, come back and VxAdmin is still configured

See apps/backend-integration-testing/src/bmd_ballot_cycle.test.ts