Use ConfigParser for High-Level Configuration

Author

Background

We have support for high-level and low-level configuration. Low-level configuration is used by developers to configure how components are wired together. We currently use ZCML for this. ZCML has it's strengths and weaknesses. It (actually, it's use in Zope 3) is continuously being simplified. ZCML is not the topic of this proposal and I would greatly appreciate it if discussion of this proposal did not include a discussion of the pros and cons of ZCML.

High level configuration is used by adminstrators or users installing Zope. It incudes things like database and log file paths, server port numbers, and so on. We currently use ZConfig for high-level configuration.

The ZConfig format is fairly simple. It is based on the Apache (and Squid) web-server configuration languages.

Problem

ZConfig is too hard to extend. It tends to cause significant coupling. For example, the configuration schemas of zdaemon, ZODB, the web server and the application server have to be combined in a single ZConfig schema, causing these separate systems to be tightly coupled.

A goal of ZConfig was to allow separate applications to share a single configuration file. This could be a boon to the person installing a system with multiple services, because all of their high-level configuration options would be in one place. We see this partially realized in Zope with zdaemon, ZODB, server, and application options in one file, however, we don't really have a way for separate applications to use the file without merging all of the application schemas, For example, the zdaemon controller process has to load the combined zdaemon, Zope, Twisted and ZODB scemas just so it can get the bits it needs from the configuration file.

The configuration of the Zope server is a rat's nest of interdependencies. The introduction of the Twisted server made this worse. It will be a major pain to package this in a sane way.

Proposal

I think we need to revisit the way high-level configuration is done. At a minimum, we need to reimplement ZConfig in a way that allows schemas to be defined more easily and in a more modular fashion. I'd really like, however, to realize the goal of allowing multiple independent applications to use the same configuration file.

We (ZC) have been evolving a throw-away prototype "buildout" system for automating creation of complete systems including Zope application servers, ZEO servers, LDAP servers, etc. Eventually, when we build the non-prototype version, we'll release this. Benji York did much of the design of this system and he decided to use the ConfigParser Python standard module to read configuration files, and thus define the configuration format. This has worked out very well. Much better than I expected.

ConfigParser takes a very different approach from ZCML and ZConfig. The parser is extremely simple and has no application knowledge. It just slurps data from one or more files and creates a simple data structure that is, essentially, a dictionary of dictionaries. Applications pull data from the configuration data structure as needed. It is up to the application to convert and validate the data it reads. Of course, it could use various frameworks to help it with this, but this is still separate from the configuration file handling.

The key difference between the ConfigParser approach (as I envision it) and the approach taken by ZCML and ZConfig, is that, with ConfigParser, applications call the configuration system to pull data, whereas, with ZConfig and ZCML, the configuration system calls the application. I think that having the configuration system call application code has worked fairly well for ZCML. I don't think it has worked well for ZConfig.

Like ZConfig, the format used by ConfigParser is familiar. ConfigParser files normally look like INI files used by many applications, especially on Windows. The ConfigParser format is by far the dominant configuration format used by Python applications.

I'd like to switch to using ConfigParser and it's format for high-level configuration. This will provide a number of benefits:

Easier management of configuration schemas. Basically each application or framework will provide it's own mechanisms for loading it's configuration data. There will no longer be a need for a centralized schema.
Data for multiple applications can be stored in a single file. For example, ZEO server and application server configuration can be included in a single file.
Because we'll be using the same configuration format used by many other Python frameworks, we'll be able to more easily interoperate. For example, we could use Paste Deploy to control integration of our applications with a server and with other WSGI applications. This will allow us to more readily leverage WSGI web components.
One fewer framework (ZConfig) to maintain. Of course, it is likely that we'll end up creating new frameworks around the new format, this offsetting some of this benefit.

ConfigParser supports reading multiple files. Options from multiple files are merged. We will, for our applications, also adopt the paste.Deploy convention of using "use" options to allow one section to extend another. For example, several application instances could share a common set of options, differing only in the port used.

Risks

The greatest challenge with this proposal is backward compatibility. We will need to support people's existing configuration files. We'll need some way to convert existing zope.conf files to ConfigParser format. We could do this at run time, so that we could support the existing format, at least for a while. See the "Conversion" section below.

One of the reasons for choosing the ZConfig format was that it allowed multiple levels of nesting. This can be achieved by having sections refer to other sections. The conversion scheme described below illustrates how this can be done.

To allow multiple applications to share a single configuration file, it will be necessary to be able to tell applications which configuration file, and section to use. This is something we can engineer into our applications.

We will need to convert the existing code to read ZConfig format to read ConfigParser format instead. For the most part, this will entail converting many existing handlers to take ConfigParser objects and section names.

Conversion

We'll need to convert existing ZConfig files to ConfigParser format. We will provide a tool for performing the conversion manaually. If Zope is started with a file ending in ".conf", the file will be converted on the fly. (ConfigParser formatted configuration files will use the ".ini" suffix.)

The conversion will use some simple rules:

Options will be converted by adding "=" between option names and values.

ZConfig sections will be converted to ConfigParser sections

ZConfig root-level options (options not in a section) and top-level sections will be collected in an application-defined section. Zope will collect these in a section named "zope" by default.
It will also be possible to tell a Zope instance to use a different section, thus allowing multiple Zope instances to be configured in a single configuration file.
Nested sections will have names created by prefixing section names with parent section names and a slash.
A nested section will have an option in the containing section of the same name and a value equal to the name of the nested section:

For example:
```
        <zodb>
          <filestorage>
            path Data.fs
          </filestorage>
        </zodb>
```
will be converted to:
```
        zodb = zope/zodb

        [zope/zodb]
        filestorage = zope/zodb/filestorage

        [zope/zodb/filestorage]
        path = Data.fs
```

Repeated sections will have numeric suffixes added to make them unique.

For example:

        <accesslog>
          # This sets up logging to both a file (access.log) and to standard
          # output (STDOUT).  The "path" setting can be a relative or absolute
          # filesystem path or the tokens STDOUT or STDERR.

          <logfile>
            path access.log
          </logfile>

          <logfile>
            path STDOUT
          </logfile>
        </accesslog>

will be converted to:

        accesslog = zope/accesslog

        [zope/accesslog]
        # This sets up logging to both a file (access.log) and to standard
        # output (STDOUT).  The "path" setting can be a relative or absolute
        # filesystem path or the tokens STDOUT or STDERR.

        logfile = zope/accesslog/logfile
                  zope/accesslog/logfile.1

        [zope/accesslog/logfile]
        path = access.log

        [zope/accesslog/logfile.1]
        path = STDOUT

Note that the logfile option in the accesslog section has multiple values, because there are repeated logfile sections.

Section names will be converted to zconfig-section-name options.

Here is an extended example. The following ZConfig file:

    # This is the configuration file for the Zope Application Server.

    %define INSTANCE  /home/jim/sample_instance

    %define CONFDIR   $INSTANCE/etc
    %define DATADIR   $INSTANCE/var
    %define LOGDIR    $INSTANCE/log

    # identify the component configuration used to define the site:
    #
    site-definition $INSTANCE/etc/site.zcml

    # number of bytecode instructions to execute between checks for
    # interruptions (SIGINTR, thread switches):
    #
    interrupt-check-interval 200

    <server HTTP0>
      type HTTP
      address 8080
    </server>
    <server HTTP1>
      type HTTP
      address 8081
    </server>
    <server>
      type PostmortemDebuggingHTTP
      address 8082
    </server>

    <zodb main>
      <filestorage>
        path $DATADIR/Data.fs
      </filestorage>
    </zodb>
    <zodb a>
      <filestorage>
        path $DATADIR/A.fs
      </filestorage>
    </zodb>

    # Access log

    <accesslog>

      <logfile>
        path $LOGDIR/access.log
      </logfile>

      <logfile>
        path STDOUT
      </logfile>

    </accesslog>

    <eventlog>

      <logfile>
        path $LOGDIR/z3.log
        formatter zope.exceptions.log.Formatter
      </logfile>

      <logfile>
        path STDOUT
        formatter zope.exceptions.log.Formatter
      </logfile>
    </eventlog>

    devmode off

Will be converted as follows:

    [DEFAULT]
    INSTANCE = /home/jim/sample_instance
    CONFDIR = %(INSTANCE)s/etc
    DATADIR = %(INSTANCE)s/var
    LOGDIR = %(INSTANCE)s/log

    [zope]
    # This is the configuration file for the Zope Application Server.

    # identify the component configuration used to define the site:
    #
    site-definition = %(INSTANCE)s/etc/site.zcml

    # number of bytecode instructions to execute between checks for
    # interruptions (SIGINTR, thread switches):
    #
    interrupt-check-interval = 200

    server = zope/server
             zope/server.1
             zope/server.2

    zodb = zope/zodb
           zope/zodb.1

    # Access log

    accesslog = zope/accesslog

    eventlog = zope/eventlog

    devmode = off

    [zope/server]
    config-section-label = http0
    type = HTTP
    address = 8080

    [zope/server.1]
    config-section-label = http1
    type = HTTP
    address = 8081

    [zope/serve.2]
    type = PostmortemDebuggingHTTP
    address = 8082

    [zope/zodb]
    config-section-label = main
    filestorage = zope/zodb/filestorage

    [zope/zodb/filestorage]
    path = %(DATADIR)s/Data.fs

    [zope/zodb.1]
    config-section-label = a
    filestorage = zope/zodb.1/filestorage

    [zope/zodb.1/filestorage]
    path = %(DATADIR)s/A.fs

    [zope/accesslog]

    logfile = zope/accesslog/logfile
             zope/accesslog/logfile.1

    [zope/accesslog/logfile]
    path = %(LOGDIR)s/access.log

    [zope/accesslog/logfile.1]
    path = STDOUT

    [zope/eventlog]

    logfile = zope/eventlog/logfile
              zope/eventlog/logfile.1

    [zope/eventlog/logfile]
    path = %(LOGDIR)s/z3.log
    formatter = zope.exceptions.log.Formatter

    [zope/eventlog/logfile.1]
    path = STDOUT
    formatter = zope.exceptions.log.Formatter