Zope 3 Packaging Proposal

Packaging of Zope 3 file-system-based software

Status

Author

Problem

An important feature of Zope 3 is that it provides a highly flexible and modular architecture for creating applications. It is hoped that Zope components will be arranged in a variety of ways to meet various application needs. (Certainly, at least one configuration will be an web application server recognizable as Zope. :)

To support this flexibility, we'd like to provide a fine-grained packaging system that supports:

Easy creation of software distributions to meet a variety of needs. For example, we'd like to make it easy to support both stable and development Zope distributions. The Zope X3.0.0 distribution will include only features that are either very stable or needed for file-system-based development. There are many other components in the Zope 3 repository that would be useful to people willing to work with still-under-development software. We'd like to be able to easily generate alternative "development" distributions with these components.
Flexible update and maintenance of distribution components

This document, which will eventually become a proposal, proposes some goals for selection and or design of a packaging system for Zope.

Goals and Assumptions

Support fine-grained software distribution packaging.
Software distribution packages will often correspond directly to Python packages that are rather small. For example, a small software distribution package might include a single Python package with only a few source files.

Note that supporting fine-grained distribution packages requires a packaging system that supports (requires) and leverages information about package dependencies.
Work with distutils. We don't want to reinvent the wheel. OTOH, we may have to work with the rest of the Python community to improve the wheel we have to provide or support package management. The most important need being dependency management. See PEP 262.
We should strive to have needed distutils improvements ready for inclusion in Python 2.4.
Work with existing platform packaging systems, most notably, RPM, Debian packages, and Fink.
Support platforms without their own packaging systems, most notably Windows.
Support easy creation of various Zope distributions. These distributions will allow simple installation of a collection of packages easily without use of a package system by end users.
Liberate the Zope repository from having to reflect distribution or packages status (e.g. core or not core.)
It should remain possible to check out the Zope 3 CVS module, execute a simple build command and run Zope in place. It should be easy to modify software, run tests and Zope and check in changes without having to resort to separate "install" steps after each change.
Provide a simpler repository layout to make Zope 3 software easier to understand and navigate.
Provide greater visibility to software dependencies.

Non Goal

An important non-goal is Zope instance creation. The software packaging mechanism will be responsible for installing software. It will generally not be responsible for setting up Zope instances. Rather, among the software installed will be software for creating Zope instances.

Jargon Problems

We use the term "package" to refer both to Python packages and to units of software distribution. There will often, but not always be a one-to-one correspondence between software distribution packages and Python packages. Distutils attempts to solve this by referring to software distributions as "module distributions". I (Jim) find this rather unsatisfying. This is an issue that will, ultimately, need to be resolved by the Python community. We will follow their lead.

Definitions

module distribution: A collection of Python modules distributed together as a single downloadable resource and meant to be installed en masse. Examples of some well-known module distributions are Numeric Python, PyXML, PIL (the Python Imaging Library), or mxBase. (This would be called a package, except that term is already taken in the Python context: a single module distribution may contain zero, one, or many Python packages.) (defined in the distutils documentation)
collection distribution: A distribution that exists primarily to express a collection of other distributions through dependencies. May include modules or other files, such as documentation or utility scripts.
application distribution: A distribution that installs an application rather than a set of packages. This distribution does not use the normal distutils setup.py interface, but provides a more conventional configure / make / make install installation pattern (for Unix), or perhaps a platform-specific installer application. This will install the software into an application home (what Zope 2 called the "Zope home").
package distribution: A distribution that is based on a package, package-contained meta data, and some of the package's sub-packages. This is a package-centric special case of "module distributions", as defined by the distutils documentation. In particular, we will make more assumptions than does distutils, about the arrangement of the information that defines a package distribution.

Definitions of related general Python terms and distutils-specific terms can be found in the distutils documentation.

Possible Vision

Work with the Python community at large to extend distutils to manage information necessary for packaging and create a simple package-management system for Python.
From the above system, provide utilities for automatically generating platform (RPM, Debian, etc.) packages.
Working within the distutils framework should allow us to support whatever distutils is willing to support, without having to worry directly about specific platforms. Our experience so far is that we deploy software that's as isolated from the rest of the system as possible; so deploying via the host platform's packaging system shouldn't be a priority.
Create a system for generating distributions from packages. For example, a source distribution might be defined by specifying a root package and satisfying it's dependency. A source distribution might then be simply a collection of gathered source packages and glue utilities (obviously based on distutils) that build and install the collected packages.
Note that source and binary distributions may not look anything like the CVS repository.

Actors

Package developers: People who work on individual software packages
Distribution Developers: People who create releases from collections of packages.
Installers: People who install distributions
Users: People who use software installed by the consumers Expectations of users vary by platform
Binary packagers: People who create binary distributions from source distributions

Use Cases

Non-developer user installs an application. They don't care that the application uses Python.
(Similarly: non-developer user installs an application. They don't care that the application uses some foundation, or that the foundation uses Python.)
- Their experience should not depend on whether some versions of Python or zope are already installed.
A developer installs an application foundation in a development area so that they can use it to develop their own applications.
A developer wants to install an application with their own application
A developer checks out an application foundation, and builds it so they can work on it or so that they can develop applications based on the most recent version of it.
Installer installs packages into an existing installation.
A distribution packager includes optional packages.
An evaluator inspects the sources without installing.
An installer or developer uses a distribution to get access to a package that is included in the distribution and not distributed separately.
- They want to know what the dependencies are. For example, if someone is going to install zope.interface, they want to be able to find out that zope.testing will be installed too.
An installer uninstalls a package or application
An installer updates a package or application

Distutils Issues

We are aware of a number of issues using distutils to package Zope software. These are collected here so we can seek to address them in this project:

Distutils has no notion of an "application" that consists of materials (data files, scripts, modules, packages, whatever) that doesn't get installed in a (potentially) shared portion of the system. On Unix, this can be similated by using install --home, but there's no way to deal with this on Windows at all.
This is particularly important for large applications such as Zope, but Zope is not the only example application.

For Unix, another aspect is that distributions should provide a more conventional interface to the person building and installing the software, such as the common ./configure / make / make install dance.

For Windows, being able to specify an installation directory other than the Python installation would go a long way to making distutils easier to use for this application.
Installation schemes should be extensible. Part of a Zope 3 installation contains ZCML files. Third-party Zope 3 components are going to want to install additional ZCML files into that location as well. In order to allow installation of third-party components into either a Zope 3 software installation or into a Zope 3 "instance home", using the same distribution, however, would require naming that location in some abstract way since the location of that directory in a software installation and in an instance home uses a different path relative to the prefix.
The expectations of "scripts" on Unix and Windows are very different. Many Unix users prefer that scripts provided for use from the command line have no extension, while Windows users actually require an extension so that Windows will know what to do with them.
Renaming the scripts to support the target platform's expectations is one approach to dealing with the issue, but may not be the right approach. Another to consider would be to allow different scripts to be provided for different platform.

The right approach to cross-platform scripts is not clear.
Lack of dependency data needed to support a packaging system. See PEP 262.
No support for package data files. Zope makes extensive use of Python packages to include files in addition to Python modules, including:
- Source files for other languages (e.g. ZPT and DTML)
- Configuration files and schema
- Images
- Test data
Distutils has minimal support for installing data files, but has no notion of data files that are in and that should be installed with packages. Phillip Eby's setuptools provides a nice way to deal with data files in packages.
Insufficient support for container packages. We sometimes want to distribute subpackages such as subpackages of zope, zope.app, com.zc, etc, but distutils doesn't warn the user if an outer package that is not provided by a distribution isn't actually a package (it just creates the directories and copies the files into place).
Issues with generating (or generated) platform packages (such as RPMs).
This space for Tres to fill in. :)

Other Issues

We need to be able to manage the packages separately from the Python installation. Packages will be installed into the "Zope home" or the "instance home".

Prototype

Fred Drake is working on a prototype command-line tool called zpkg, that can create collection, application, and package distributions. More information is available in the doc/ directory of the software package, or you can read an easier-to-navigate version formatted in HTML.

The README.txt Problem: This discusses an issue regarding the user experience when unpacking a distribution created using the packaging tool. The problem has been solved in zpkg by enhancing the metadata to allow the distribution root to be controlled directly.

Proposal

None yet (in spite of substantial progress on the prototype tool) Docs: unknown location