Use Zope 2.7.3 or later
The hardest thing about Zope unit tests was, and sometimes still is, running them. It can be a terrifying black art.
In Zope 2.6 and earlier, you had a test runner for products in lib/python/Products, another for CMF products and you needed a third-party patch for those in the standard Products-directory. Zope 2.7.0 introduced a new test runner, test.py but retained the old without deprecation warnings, and introduced more obscure bugs, making this even more confusing and terrifying. But slowly during the life of Zope 2.7 things have become better. See the troubleshooting section below if you get stuck.
So, use Zope 2.7.3 or later. You can use the test.py from latest Zope with earlier versions of Zope 2.7 but you will need to know what you are doing. Earlier versions of test.py aren't useful at all.
Testing one or more products
Usually you want to run unit tests for one or more products. The basic command is:
zopectl test
This will run all unit tests in your Products directory. zopectl is found in INSTANCE_HOME/bin; it runs the bin/test.py testrunner, but adds some useful defaults:
-v increases verbosity. You can add -vv for even more output. --config-file etc/zope.conf reads in the Zope configuration file,so that important paths are set up. --libdir Products tells test.py to include Products as a module path and to include the tests there.
NB if you have several zope instances, you must use the zopectl from the instance where your product is.
zopectl doesn't run on windows, so there you need to spell it out:
c:\path\to\python.exe bin\test.py -vC etc\zope.conf --libdir Products
Perhaps it's simpler after all to make sure SOFTWARE_HOME/bin/test.py is in your path and just do:
cd myproductdir; test.py --libdir . -v
To see the rest of the command options you can run it with --help:
zopectl test --help
Quite often you don't want to run all the tests in Products, either because it takes too much time, or because in the real world you may have installed third-party products that fail to clean up after themselves and thus cause other tests to break; see Troubleshooting below for more information about this. You can limit the search for tests to a specific directory, with the --dir parameter:
zopectl test --dir Products/CMFCore/tests/
You can also add different filter statements to the command, to specify it even further. This command will run only test in the testCookieCrumbler.py file:
zopectl test --dir Products/CMFCore/tests/ testCookieCrumbler.py
And this will run only the test called testCookieLogin, in the testCookieCrumbler.py file:
zopectl test --dir Products/CMFCore/tests/ testCookieCrumbler.py testCookieLogin
The test output should look something like this:
Running unit tests at level 1 Running unit tests from /home/regebro/Zopes/Zope-trunk/Products/CMFCore/tests Parsing /home/regebro/Zopes/Zope-trunk/etc/zope.conf ..................... ---------------------------------------------------------------------- Ran 21 tests in 0.130s OK
Testing Zope
If you want to run the unit tests for Zope itself, then follow this procedure:
- make a svn checkout of the Zope version in question, cd to that directory
- run
./configure && make inplaceto build Zope in-place (which nowaday only Zope-devs do) - run
python test.py(where python should be the same python you previously compiled that Zope with)
Run tests with a GUI
(Not tried recently) You can run the test with a GUI as well. You need the Zope 3 unittestgui.py in your python path or current directory. (The standard one in python-unit complains). Then:
/zope1/Products/ZWiki$ PYTHONPATH=/zope/lib/python SOFTWARE_HOME=/zope/lib/python \ INSTANCE_HOME=/zope1 python /zope/test.py -m --libdir . Running unit tests from /zope1/Products/ZWiki/. Loading Zope, please stand by ... done (1.028s) Installing ZCatalog ... done (0.026s) Installing ZWiki ... done (0.094s)
You should see the unittestgui window. Click the > button to unminimize it, or double-click the progress bar to run tests.
See Also
- Test Runner Comparison compares available test runners for Zope 2
Troubleshooting
There are still plenty of pitfalls for a zope tester. Here are some problems and solutions.
ZopeTestCase not installed:
ImportError: cannot import name ZopeTestCase
Many products use the ZopeTestCase test framework these days. This was not included with Zope until Zope 2.8. If you are using Zope 2.7, it must be installed (as SOFTWARE_HOME/lib/python/Testing/ZopeTestCase) before you can run tests.
wrong zopectl:
/zope2/Products/HamCannon$ zopectl test --libdir $PWD Running tests via: /usr/bin/python /opt/Zope-2.7/bin/test.py -v --config-file /zope1/etc/zope.conf --libdir /\zope2/Products/HamCannon Running unit tests at level 1 Running unit tests from /zope2/Products/HamCannon Parsing /zope1/etc/zope.conf ---------------------------------------------------------------------- Ran 0 tests in 0.000s OK
Above, zopectl is coming from another zope instance (/zope1/bin/zopectl). Be sure to run the zopectl from the instance where your product is.
old test code:
You need to set the ZOPEINSTANCE environment variable to refer to the root of your zopeinstance
or:
AttributeError: 'module' object has no attribute 'test_suite'
or sometimes:
Ran 0 tests in 0.000s
The product contains old test framework code. Look for code mentioning "framework" at the top and bottom of the tests/test* files. Also look for helper files like tests/runalltest.py, tests/framework.py etc. These are for earlier test runners and confuse the modern one; the product should be updated.
ZopeTestCase suite is missing an __init__.py:
Ran 0 tests in 0.000s
You can easily identify a ZopeTestCase test suite by looking at what it imports; ZopeTestCase should be imported in the source. (Or PortalTestCase?, or CMFTestCase?, or PloneTestCase?, or CPSTestCase?...) Also, there is usually a framework.py file in the tests/ directory. If __init__.py is missing from the tests/ subdirectory, the test runner will not be able to find the tests. The product should be be fixed: simply add an empty __init__.py to its tests/ subdirectory (and notify the product author).
tests.py in a subdirectory:
File "/usr/local/src/Zope-2.7.3-0/bin/test.py", line 482, in package_import
mod = __import__(modname)
ImportError: No module named subdir.tests
A file named tests.py in a subdirectory can confuse the test runner. Rename it.
TextIndexNG2 initialisation problem:
TING2 sometimes fails to initialise during testing. Timing related ? It looks like this:
$ zopectl test --libdir .
Running tests via: /usr/bin/python /usr/local/src/Zope-2.7.3-0/bin/test.py -v --config-file /zope1/etc/zope.conf --libdir .
Running unit tests at level 1
Running unit tests from /repos/ZWiki
Parsing /zope1/etc/zope.conf
Installing ZCatalog ... done (0.023s)
Installing ZWiki ... done (0.105s)
Installing TextIndexNG2 ... Traceback (most recent call last):
File "/usr/local/src/Zope-2.7.3-0/bin/test.py", line 918, in ?
process_args()
...
File "/usr/local/src/Zope-2.7.3-0/lib/python/OFS/Application.py", line 738, in install_product
initmethod(context)
File "/repos/ZWiki/__init__.py", line 87, in initialize
# How to handle versions & upgrades of imported content nicely ?
File "/repos/ZWiki/__init__.py", line 23, in ?
def initialize(context):
File "/usr/lib/python2.3/popen2.py", line 90, in wait
pid, sts = os.waitpid(self.pid, 0)
OSError: [Errno 10] No child processes
I've been told by Andreas that this is a zope 2.7.3 zopectl and earlier versions bug. It should be fixed in 2.7.4. If you need to run 2.7.3 like I do, I have fixed it manually by changing converters/__init__.py, line 23 : PO.wait() protect it from OSError? with a try..except OSError? Tarek.
ImportError?: No module named cPersistence (on debian):
You may be seeing debian zope2.7 bug #305854 :
$ zopectl test -v --libdir .
Running tests via: /usr/bin/python /usr/lib/zope2.7/bin/test.py --config-file /zope1/etc/zope.conf -v --libdir .
Running unit tests at level 1
Running unit tests from /repos/ZWiki
Parsing /zope1/etc/zope.conf
No module named cPersistence
...
File "/usr/share/zope2.7/lib/python/Zope/Startup/datatypes.py", line 18, in ?
from ZODB.config import ZODBDatabase
File "/usr/share/zope2.7/lib/python/ZODB/__init__.py", line 19, in ?
import cPersistence, Persistence
ImportError: No module named cPersistence
See that page for an explanation and workaround (remove a realpath call from /usr/share/zope2.7/bin/test.py).
ImportError?
Some third-party products may have test code that depends on modules that the product itself does not depend on. E.g. tests may test optional features that interact with yet another product.
If you are a product author, you should take care to try running your product tests with ONLY your product and its required dependencies installed.
Tests run OK in isolation, but give errors when all tests are run
If you use zopectl test to run all your Product tests at once, you may encounter large numbers of errors from tests that work fine when you run them individually. This is a sure sign that some test module is not cleaning up after itself and thus breaking some setup code in a subsequent test. A typical error in this case is BadRequest? ("The id is already in use"). As a wise man once said on the zope mailing list:
That's how reality looks. It's unfortunate, but there ya go.
If you are a product author, please take care that your test suite leaves no trace of itself, and it's a good idea to try running your tests alongside many other common products. If you find that the problem is caused by somebody else's product, check if a more recent version fixes the problem and if not, let the author know about the problem.
Also note that ZopeTestCase test suites do not mix well with non-ZopeTestCase suites. As Stefan Holek (the ZopeTestCase author) pointed out:
ZTC-based tests don't play nicely with non-ZTC tests in oh so many ways. Party because ZTC assumes it has full control over the Zope environment, and party because those other tests do the same ;-) I don't really feel like tackling this at all. As a rule of thumb you can run all ZopeTestCase (CMFTestCase, PloneTestCase) tests in your instance in one big suite without problems, but you can't mix them with, say, the CMF tests. Now, come Zope3 and Five, ZTC has served its purpose and is on its way out anyway.
In the end, for running tests of Zope 2 products, it may be simpler and more reliable to write a simple shell (or windows .bat, or python) script that lists all the products you want to test and runs each test suite individually.
On the other hand, core Zope unit tests should never cause other core tests to break. In the rare case that you should discover one, file a bug report in the Zope Collector.
2.8 notes --simon, Thu, 23 Jun 2005 02:12:19 -0700 reply
From Stefan Holek:
You may have noticed that when running tests under Zope 2.8, all log messages are sent to stdout by default, spamming your terminal. You can avoid this by:
a) Copying log.ini from a Zope 2.8 sandbox to the instance home
b) Using test.py to run the tests, e.g. ./bin/zopectl test --libdir Products/CMFPlone
Also note that by default test.py scans for stale *.pyc files, which is time-consuming and not necessary most of the time. Adding the --keepbytecode flag to the command line skips this scan and saves a lot of time, especially under Zope 2.8 (which includes all of Z3 -> many files to check).:
./bin/zopectl test --keepbytecode --libdir Products/CMFPlone
using test.py does not help. -- Tue, 30 Aug 2005 08:34:16 -0700 reply
You need the log.ini.
Contradictory advice: framework.py and ZopeTestCase --slinkp, Tue, 04 Oct 2005 09:08:16 -0700 reply
"look for helper files like tests/runalltest.py, tests/framework.py etc. These are for earlier test runners and confuse the modern one; the product should be updated." And yet earlier on this page, we were told that "Most products use the ZopeTestCase test framework these days." And ZopeTestCase suites are supposed to contain a framework.py, and these do not seem to cause a problem with test.py. So there's contradictory advice here.
test.py not installed --simon , Wed, 08 Feb 2006 16:20:00 -0800 reply
With Zope 2.9, you need to copy $ZOPE_SRC/test.py to $SOFTWARE_HOME/bin (make install doesn't do this - cf http://www.zope.org/Collectors/Zope/1989).
Then you can run tests with eg:
$INSTANCE_HOME/bin/zopectl test -m Products.MyProduct
2.9 updates - this should be folded in --slinkp, Tue, 21 Mar 2006 07:22:24 -0800 reply
http://www.dataflake.org/docs/testrunner_madness.html
Five --pigletto, Fri, 17 Nov 2006 10:46:31 -0800 reply
For testing products with Five take a look at Five/testing/README.txt
recent useful list messages --slinkp, Wed, 13 Jun 2007 17:38:00 +0000 reply
here's how to test particular packages that are not part of your instance home:
http://mail.zope.org/pipermail/zope-cmf/2007-June/026161.html
http://mail.zope.org/pipermail/zope-cmf/2007-June/026162.html