Unified test runner for sitescripts

[kvas]

Background

We have several tests lying in test folders around sitescripts. They cover different modules and have different dependencies. It appears that they were not typically run together and there has been no attempt to automate the running of the tests of measure coverage.

If we had one centralised place to run the tests for a particular subproject of sitescripts or all the tests in a more automatic way it would give us some advantages:

• It would be easy for anyone to run the tests without trying to figure out the implicit dependencies, test locations, etc.
• When adding new code, or changing existing code that has no test coverage, there would be an obvious documented way to add tests for it.
• If the work on one part of sitescripts breaks something else by side-effect, it would be less likely to go unnoticed.
• We could generate a coverage report and know what is covered and what is not.
• It would be easier to attach the centralised test runner to some kind of continuous integration system to make sure the test failures are immediately caught.

I think it makes sense to use ​Tox for this. Also since sitescripts is more like several modules than like a single module, we would like to be able to run the tests of different modules independently.

What to change

• Add tox.ini to sitescripts with one environment per submodule (include only submodules that have tests).
• For each environment specify the list of dependencies that are necessary to run the tests of the submodule. Configure running the existing tests of the submodule.
• Tox should be able to run the tests without installing sitescripts in the virtualenvs that it creates (sitescripts doesn't have setup.py so this will not work). This can be achieved by adding skipsdist=true to tox section of tox.ini.
• Set envlist in tox.ini to the list of submodules whose tests don't have any dependencies that cannot be managed by Tox (like a database server, for example). Running tox without arguments should run all the tests that are environment-independent.
• Document in README.md how to run the existing tests and how to add new tests.

[kvas]

Run tests of different submodules separately: this is more practical for how we actually work on parts of sitescripts. Separate the Tox environments by submodule and don't use setup.py for installing sitescripts into the virtualenvs.

[sebastian]

Any reason to exclude tests that require a database? Sure tox cannot automatically make sure the database server is installed and configured, but at least you don't have to bother about Python dependencies, PYTHONPATH, etc. if running those tests.

Also mind adding, that we don't want to create setup.py scripts for tox but rather configure it to work without? It seems simply setting skipsdist=true does the trick.

[abpbot]

A commit referencing this issue has landed:
​Issue 3985 - Unified test runner for sitescripts