CMS testing automation

[kvas]

Background

The test suite of our CMS covers the basics but unfortunately it's still far from comprehensive. A more reliable way to ensure that a change doesn't break anything is to generate our websites with both known good version of CMS and the version with the change and then compare the outputs. I've done this a few times and it's rather annoying to set everything up. Since the test suite is not expected to be comprehensive any time soon (and I'm not even sure what this means anyway, as the expected functionality is not documented) it seems prudent to have a script that would automate testing on our real websites.

What to change

Implement a script that would take a list of websites, generate them with two revisions of CMS and then compare the results. The script should accept options for:

• Python interpreter to use for running the CMS.
• Location of CMS repository.
• Base revision (default to master).
• Test revision (default to current revision).
• Location and the list of website repositories to test.
• Directory for storing temporary files (default to TMPDIR).

Note: Since generation of some websites takes a while, and switching between revisions in Hg is fast, it seems more prudent to go site by site (rather than generate everything with base revision, then everything with test revision). This way any possible errors will be discovered sooner.

Note: It also make sense to reuse content generated earlier when possible. If we pin the revision hashes of both CMS and the website, the output should be the same, so it can be used again without the need to regenerate.

Note: The script should not modify the working copies of websites or of CMS. In particular for the CMS repo, it seems best to clone it to a temporary directory and do revision switching there. It will also ensure that local changes will not affect the testing.

[abpbot]

A commit referencing this issue has landed:
​Issue 5934 - CMS testing automation