Reunite Zope 2 and Zope 3 in the source code repository

Author

Philipp von Weitershausen, philikon@philikon.de

Status

IsProposal

Original

http://codespeak.net/svn/user/philikon/ReuniteZope2AndZope3.txt

Current situation

Zope 2 lives at svn://svn.zope.org/repos/main/Zope. Zope 3 lives at svn://svn.zope.org/repos/main/Zope3. Right now, Zope 2 includes nearly everything from the Zope 3 repository, even though it doesn't use everything (but most of it) in a distribution.

Both Zope 2 and Zope 3 also use the following extra technologies which are inserted via 'svn:externals':

• ZODB
• ZConfig
• zdaemon
• zpkg
• zope.testing (yes, this is an extra technology because it's managed within neither Zope repository tree)
• docutils (external project, svn.zope.org isn't the primary repository; it's also not included as an external, instead Zope 2 and Zope 3 manage two different copies which are not necessarily in sync.)

Problems

• Less than optimal use of synergies:

  Recent improvements to Zope 2 should have been forward-ported to Zope 3 as well. I'm thinking of the improved Windows service support and the file stream iterator, for example. Instead of having things implemented twice in the future, they should only be implemented once (preferably in below the zope namespace) and simply used twice.

• Zope 2 needs Zope 3 to be more flexible:

  Zope 2 is using a lot of Zope 3 technology through Five, Zope 2.9 even started using some in the core (e.g. the test runner, events, i18n). Zope 2.10 will go even further and probably use things like zope.filereference for ZIP file import support, zope.pagetemplate for ZPTs? and possibly other things. More use of Zope 3 technology in Zope 2 will inevitably mean some refactorings on the Zope 3 side. One part where this is already evident are the ZCML directive handlers; others may apply as we find out about them.

• Zope 3 refactorings affect Zope 2:

  When Zope 3 is refactored, the parts that depend on it in Zope 2 (and Five) should also be refactored. This will avoid getting Zope 2 and Zope 3 out of sync regarding the use of Zope 3 technology. Small refactorings on the Zope 3 side have already broken tests in Zope 2 or Five in the past.

• The zope namespace should be open for Zope 2 developers:

  It is generally agreed by now that top-level packages are a bad idea and that we should all use the zope package namespace. We already do for Zope 3, but we should start doing so for Zope 2 as well (zope.testrecorder is some nice pioneer work in that respect). Having to go to the Zope 3 tree to put a package there and then later stitch it in via an svn:externals entry adds more development discomfort than necessary (e.g. for experimental stuff that would live on a branch, one would have to create two branches, etc.). In short, people will be discouraged to put their stuff in the zope package namespace. Instead they'll continue to create stuff in Products, something which should be discouraged soon, if not now.

Proposal

Manage Zope 2 and Zope 3 in the same SVN repository tree.

My suggestion is to use the Zope 3 repository layout for the new tree (I prefer a src directory over lib/python). For that we move all the Zope 2 stuff to the Zope 3 tree. Then we delete the Zope tree and rename the Zope3 to Zope. In pseudo shell commands:

  $ svn mv Zope/lib/python/* Zope3/src
  $ svn rm Zope
  $ svn mv Zope3 Zope

Note: Zope 2 and Zope 3 both contain separate copies of docutils and RestrictedPython. The former should always be in sync because it's a vendor import. In the latter case, the package was copied from Zope 2 to Zope 3 in rev 26688 and has been edited in both branches, in the Zope 2 branch more than in the Zope 3 branch. Before this proposal is implemented, the changes from one branch should be merged to the other so that there's only one authorative RestrictedPython implementation.

Your questions answered

This is a change too huge for me!

I doubt that. Look at a checkout of Zope 2.8 or higher. It essentially is a checkout of both Zope 2 and Zope 3, except that the Zope 3 part is wired in via an svn:externals entry. Moving them together wouldn't change so much for the Zope 2 developer.

Sure, the Zope 3 developer would see a couple more packages. Packages, though, that he would normally not care about, except when running tests before a check-in.

Why do you want to pollute the clean Zope 3 codebase with Zope 2 rubbish?

I don't. I just want both codebases to live together in the same repository tree (which essentially means they'll be living together in a working copy). A compelling advantage of this is that Zope 2's evolution can benefit from the synergies of Zope 3 development. I even believe that sometimes it can happen the other around. In some minor things, Zope 2 has received more sophisticiated features than Zope 3. Windows service support and the file stream iterator are such examples.

What happens when we tag a release?

We could tag both a Zope 2 and Zope 3 release with the same tag, but we don't have to. Either way, the release schedule for their major version is synchronized anyway. That means that having one tag for them, at least in the alpha, beta and RC phase, would actually reflect what we're trying to do in the release process.

Does that mean Zope 2 and Zope 3 will be packaged as one?

They already are! Since version 2.8 the Zope 2 distribution always contains the current Zope 3 stable release. Maybe in the future, with a different packaging strategy, we won't have to package them together anymore. Instead we could have Zope 2 require a suitable Zope 3 installation already in place (possibly throuh eggs). Having Zope 3 be distributed by itself won't change.

Note: The way we organize a repository has very little to do with the way we make packages. The Zope 3 repository contains a lot more than what's packaged in a typical Zope 3 distribution. And since Zope 2 includes a full Zope 3 installation, even the parts it doesn't use (e.g. twisted) are in a Zope 2 checkout. They're not in a Zope 2 distribution, though.

What happens to stuff that isn't in Python packages?

You're probably refering to the bin, doc, utilities, skel, zopeskel et.al. directories as well as the in-place build system. Some stuff would be conflicting when the locations are thrown together. Here are some suggestions on what to do:

doc

My suggestion is to keep separate doc directories, e.g. doc2 and doc3 (note that this doesn't affect the name of those directories in the tarball; it could still simply doc). In the long term, one could think about merging the doc{2|3} directories into one, though that raises problems with standard texts which are available in both trees, e.g. CHANGES.txt. Or maybe it doesn't because we'll only have one CHANGES.txt at some point.

utilities

The contents of this directory can simply be thrown together. Zope 2's mkzopeinstance.py and mkzeoinstance.py make an exception, they should be in the in-place instance's bin directory.

setup.py, configure, Makefile, and the in-place build system

Since both Zope 2 and 3 now use zpkgutils for the in-place build, few changes if any would be necessary in setu.py. For the Makefile Zope 3 uses a simple hard-wired one while Zope 2 generates it via a configure script. This is mostly for choosing the right Python interpreter executable and setting a view variables for the in-place instance creation. At least the in-place instance generation could be made unnecessary by providing a default in-place instance like in Zope 3 (see below). The (generated) Zope 2 Makefile does a bit more than its Zope 3 equivalent, which is mostly because it was packaged in a release tarball. Most of that isn't needed anymore, though, since Zope 2 is packaged with zpkgutils (some of the details here weren't figured out in Zope 2.9 at the time of this writing, though).

bin, skel/zopeskel, and the in-place instance system

Zope 3 comes with an in-place instance which resides in the root of the Zope3 tree. Zope 2's Makefile can create an in-place instance. When living in the same repository tree, the two instances would conflict. I suggest to have two default in-place instances, one at zope2inst, one at zope3inst. They would hold whatever the in-place instance holds now, including the skel/zopeskel directories that are the skeleton for new instances as well as the mkzopeinstance/mkzeoinstance scripts. Probably the only top-level files that can remain would be related to the test runner.

Do I have to throw away my Zope 2 and Zope 3 sandboxes?

No, you don't. You will be able to switch them over to the new location. There might, however, be problems due svn:externals entries now appearing as real directories. In this case deleting the offending directory and re-initiating svn up should solve the problem.

Risks

• There will be a bigger burden on the Zope developer in several ways: more tests to run, more code to touch when refactoring, etc. Of course, the flip side is that Zope 2 and Zope 3 will be much better kept in sync than before.
• Developers of either Zope 2 or Zope 3 might find it confusing to be seeing the other projects code. This should really only affect Zope 3 developers since Zope 2 developers already see all the Zope 3 code in their checkout.
• The distinction between the two projects will be blurred from the development point of view. This will inevitably have an affect on the development process. For example, there will most probably be only one checkins mailing list and soon we'd be thinking about throwing together the development lists and issue trackers as well. Yet another logical conclusion would be to present Zope 2 and 3 equally on the community web page. Personally I think that all this could very much be a good thing, but other people might not.

Implementation progress

None so far, still in discussion.