Performance regression tool
Performance regression test tool
Status
Authors
Tarek Ziadé (tziade_at_nuxeo.com), StephanRichter
Problem/Proposal
Implement performance regression testing tools in zope testing framework.
Goals
The goal of this proposal is to enhance the test framework by giving to the test developer the ability to mark some tests that are hot spots, for a maximum execution time.
By doing it, it protects performance regressions when code is added: if a new addition of code is slowing down the test, a failure is thrown.
Proposed Solution
The best way to measure a test performance rate is to use Pystones.
- A decorator and a wrapper, that let the user mark a test for a max pystone rate
in functional test (wrapper) or in unit test (decorator)
the code can be viewed here: http://svn.zope.org/Zope3/trunk/src/zope/app/testing/?rev=39009
- A special mode in the test runner to get feedback on each non-marked test, such as:
- the actual pystones for the test
- some extra infos, like the complexity, the numbers of callees, ...
- a "how to write performance regression tests" tutorial, to teach developers the process of scaling tests, then marking hotspots.
- Benji added this kind of feature in testbrowser, and it looks pretty much to what we could have in other test context/tools.
Risks:
- open issue: the tolerance can vary when the CPU is busy on other tasks, as modules like hotshot runs tests 3 times and get the min time, we therefore need to minimize this problem. the solution at this time is to introduce the TOLERANCE thing, but may change after we get some feedback.