Integration testing of scripts in playbooks


As of now Enough assumes that all code that needs testing is in the enough directory. Part of it is exercised when tests for playbooks are run. But there currently are no tests that run against code that is found in the playbooks themselves, such as the one from SecureDrop that creates an admin user.

It could be argued that a if a script is complex enough to deserve testing, it should not be implemented in a playbook. It should either be a standalone script published elsewhere. Or be part of the enough python package and re-used via import instead. But that does not take into account the following use case which presented itself while implementing the Gitea playbook:

  • The playbook requires code that should exist in Gitea and Woodpecker to facilitate registration but does not, yet.
  • Code to cope with this shortcoming is written as part of the playbook and is a few hundred lines long

This code is short lived, it’s a patch to be removed eventually when Gitea & Woodpecker evolve, therefore it would not make sense to move it as a permanent feature of Enough or any other software.

I think the solution for that is to use the playbooks tests to also run unit tests and integration tests for the temporary script. I’m trying this as part of the development of the hostea playbook.

Here is an example: the script is installed by a role. The tests are run after the playbook completes and can use the Gitea and Woodpecker instances that it created.