Chapter 30
Local Utilities

Difficulty

Contributer

Skills

Be comfortable with the Component Architecture, specifically utilities.
Be familiar with the Site Manager Web GUI.
Know the message board example as this affects somewhat the smiley support.
You should be familiar with the global utility chapter, since this chapter creates its local companion.

Problem/Task

It is great to have our global smiley theme utilities. It works just fine. But what if I want to provide different icon themes for different message boards on the same Zope installation? Or I want to allow my online users to upload new themes? Then the global smiley theme is not sufficient anymore and we need a local and persistent version. This chapter will create a local smiley theme utility.

Solution

30.1 Introduction to Local Utilities

The semantics of local utilities are often very different than the ones for global utilities and have thus a very different implementation. For one, local utilities can be fully managed, which means that they can be added, edited and deleted. Only the first one of these actions is possible for their global counterparts. Furthermore, and most important, local utilities must know how to delegate requests to other utilities higher up, including the global version of the utility. All of these facts create a very different problem domain, which I intend to address in this chapter.

From the global smiley utility we already know that its purpose is to manage smileys (in a very limited way). Since the local smiley theme must be able to manage smileys fully, it is best to make the utility also a container that can only contain smileys. A smiley component will simply be a glorified image that can only be contained by local smiley themes. Thus we need to develop an ILocalSmileyTheme that extends IContainer. This interface must also limit its containable items to be only smileys. The second interface will be ISmiley, which simply extends IImage and only allows itself to be added to smiley themes.

Like all other local components, local utilities must be registered. One way would be to write a custom registration component; however, the simpler method is to have the local smiley theme provide the ILocalUtility marker interface. A registration component (including all necessary views) exists for any object that provides this interface making the implementation of a local utility much simpler.

30.2 Step I: Defining Interfaces

As I pointed out in the introduction, we will need two new interfaces. The first one is the ISmiley interface:

1  from zope.schema import Field
2
3  from zope.app.container.constraints import ContainerTypesConstraint
4  from zope.app.file.interfaces import IImage
5
6  class ISmiley(IImage):
7      """A smiley is just a glorified image"""
8      __parent__ = Field(
9          constraint = ContainerTypesConstraint(ISmileyTheme))

As I said before, the smiley component is simply an image that can only be added to smiley themes. The second interface is the ILocalSmileyTheme, which will manage all its smileys in a typical container-like fashion:

1  from zope.app.container.constraints import ItemTypePrecondition
2  from zope.app.container.interfaces import IContainer
3
4  class ILocalSmileyTheme(ISmileyTheme, IContainer):
5      """A local smiley themes that manages its smileys via the container API"""
6
7      def __setitem__(name, object):
8          """Add a IMessage object."""
9
10      __setitem__.precondition = ItemTypePrecondition(ISmiley)

After we make the local smiley theme a container, we declare that it can only contain smileys. If you do not know about preconditions and constraints in interfaces, please read the chapter on creating a content component.

30.3 Step II: Implementation

Implementing the smiley is trivial. In a new Python file called localtheme.py add the following:

1  from zope.app.file.image import Image
2  from interfaces import ISmiley
3
4  class Smiley(Image):
5      implements(ISmiley)

Now we just need to provide an implementation for the theme. As before, we can use the BTreeContainer as a base class that provides us with a full implementation of the IContainer interface. Then all that we have to worry about are the three ISmileyTheme API methods.

1  from zope.component.exceptions import ComponentLookupError
2  from zope.interface import implements
3
4  from zope.app import zapi
5  from zope.app.container.btree import BTreeContainer
6  from zope.app.component.localservice import getNextService
7
8  from interfaces import ISmileyTheme, ILocalSmileyTheme
9
10  class SmileyTheme(BTreeContainer):
11      """A local smiley theme implementation."""
12      implements(ILocalSmileyTheme)
13
14      def getSmiley(self, text, request):
15          "See book.smileyutility.interfaces.ISmileyTheme"
16          smiley = self.querySmiley(text, request)
17          if smiley is None:
18              raise ComponentLookupError, 'Smiley not found.'
19          return smiley
20
21      def querySmiley(self, text, request, default=None):
22          "See book.smileyutility.interfaces.ISmileyTheme"
23          if text not in self:
24              theme = queryNextTheme(self, zapi.name(self))
25              if theme is None:
26                  return default
27              else:
28                  return theme.querySmiley(text, request, default)
29          return getURL(self[text], request)
30
31      def getSmileysMapping(self, request):
32          "See book.smileyutility.interfaces.ISmileyTheme"
33          theme = queryNextTheme(self, zapi.name(self))
34          if theme is None:
35              smileys = {}
36          else:
37              smileys = theme.getSmileysMapping(request)
38
39          for name, smiley in self.items():
40              smileys[name] = getURL(smiley, request)
41
42          return smileys
43
44
45  def queryNextTheme(context, name, default=None):
46      """Get the next theme higher up."""
47      theme = default
48      while theme is default:
49          utilities = queryNextService(context, zapi.servicenames.Utilities)
50          if utilities is None:
51              return default
52          theme = utilities.queryUtility(ISmileyTheme, name, default)
53          context = utilities
54      return theme
55
56  def getURL(smiley, request):
57      """Get the URL of the smiley."""
58      url = zapi.getView(smiley, 'absolute_url', request=request)
59      return url()

Line 14-19: This implementation is identical to the global one. We have the method querySmiley() do the work.
Line 21-29: If the requested smiley is available in the theme, simply return its URL. However, if the smiley is not found, we should not give up that quickly. It might be defined in a theme (with the same name) in a level higher up. The highest layer are the global components. If a theme of the same name exists higher up, then try to get the smiley from there. If no such theme exists, then its time to give up and to return the default value.
This generalizes very nicely to all local components. Local components should only concerned with querying and searching their local place and not stretch out into other places. For utilities, the request should then always be forwarded to the next occurrence at a higher place. This method will automatically be able to recursively search the entire path all the way up. The termination condition is usually the global utility, which always has to return and will never refer to another place. If you do not have a global version of the utility available, then you need to put a condition in your local code, terminating when no other utility is found.
Line 31-42: This is method that has to be very careful about the procedure it uses to generate the result. The exact same smiley (text, theme) might be declared in several locations along the path, but only the last declaration (closest to the current location) should make it into the smiley mapping. Therefore we first get the acquired results and then merge the local smiley mapping into it, so that the local smileys are always added last. Note that this implementation makes this method also recursive, ensuring that all themes with the matching name are considered.
Line 34-43: This method returns the next matching theme up. Starting at context, the queryNextService() method walks up the tree looking for the next site in the path. If a site is found, it sees whether it finds the specified service (in our case the utility service) in the site. If not, it keeps walking. It will terminate its search once the global site is reached ( None is returned) or a service is found.
If the utilities service was found, we now need to ensure that it also has a matching theme. If not we have to keep looking by finding the next utilities service. If a matching theme is found, the while loop’s condition is fulfilled and the theme is returned.
Line 45-48: Since smiley entries are not URLs in the local theme, we look up their URLs using the absolute_url view.

As you can see, the implementation of the local theme was a bit more involved, since we had to worry about the delegation of the requests. But it is downhill from now on. What we got for free was a full management and registration user and programming interface for the local themes and smileys, which is the equivalent of the ZCML directives we had to develop for the global theme.

Until now we always wrote the tests right after the implementation. However, tests for local components very much reflect their behavior in the system and the tests will be easier to understand, if we get the everything working first. Therefore, we will next develop the necessary registrations followed by providing some views.

30.4 Step III: Registrations

First we register the local theme as a new content type and local utility. Making it a local utility will also ensure that it can only be added to site management folders. Add the following directives to you configuration file:

1  <zope:content class=".localtheme.SmileyTheme">
2    <zope:factory
3        id="book.smileyutility.SmileyTheme"
4        title="Smiley Theme"
5        description="A Smiley Theme"
6        />
7    <zope:implements
8        interface="zope.app.utility.interfaces.ILocalUtility"
9        />
10    <zope:implements
11        interface="zope.app.container.interfaces.IContentContainer"
12        />
13    <zope:implements
14        interface="zope.app.annotation.interfaces.IAttributeAnnotatable"
15        />
16    <zope:allow
17        interface="zope.app.container.interfaces.IReadContainer"
18        />
19    <zope:require
20        permission="zope.ManageServices"
21        interface="zope.app.container.interfaces.IWriteContainer"
22        />
23    <zope:allow
24        interface=".interfaces.ISmileyTheme"
25        />
26  </zope:content>

General Note: The reason we use the zope: prefix in our directives here is that we used the smiley namespace as the default.

Line 7-9: Declare the local theme component to be a local utility. Since this is just a marker interface, no special methods or attributes must be implemented.
Line 10-12: In order for the precondition of __setitem__() to work, we need to make the smiley theme also a IContentContainer. This is just another marker interface.
Line 13-15: All local components should be annotatable, so that we can append Dublin Core and other meta-data.
Line 16-18: Allow everyone to just access the smileys at their heart’s content.
Line 19-22: However, for changing the theme we require the service management permission.
Line 24-27: We also want to make the theme’s API methods publicly available.

Now that we just have to declare the Smiley class as a content type.

1  <zope:content class=".localtheme.Smiley">
2    <zope:require
3        like_class="zope.app.file.image.Image"
4        />
5  </zope:content>

Line 2-4: Just give the Smiley the same security declarations as the image. Since the smiley does not declare any new methods and attributes, we have to make no further security declarations.

The components are registered now, but we will still not be able to do much, since we have not added any menu items to the add menu or any other management view.

30.5 Step IV: Views

As you will see, the browser code for the theme is minimal, so that we will not create a separate browser package and we place the browser code simply in the main configuration file. As always, you need to add the browser namespace first:

1 xmlns:browser="http://namespaces.zope.org/browser"

Now we create add menu entries for each content type.

1  <browser:addMenuItem
2      class=".localtheme.Smiley"
3      title="Smiley"
4      description="A Smiley"
5      permission="zope.ManageServices"
6      />
7
8  <browser:addMenuItem
9      class=".localtheme.SmileyTheme"
10      title="Smiley Theme"
11      description="A Smiley Theme"
12      permission="zope.ManageServices"
13      />

We also want the standard container management screens be available in the theme, so we just add the following directive:

1  <browser:containerViews
2      for=".localtheme.SmileyTheme"
3      index="zope.View"
4      contents="zope.ManageServices"
5      add="zope.ManageServices"
6      />

Practically, you can now restart Zope 3 and test the utility and everything should work as expected. Even so, I want to create a couple more convenience views that make the utility a little bit nicer.

First, you might have noticed already the “Tools” tab in the site manager. Tools are mainly meant to make the management of utilities simpler; and the best about it is that a tools entry requires only one simple directive:

1  <browser:tool
2      interface=".interfaces.ISmileyTheme"
3      title="Smiley Themes"
4      description="Smiley Themes allow you to convert text-based to icon-based
5      smileys."
6      />

Line 1: Since tools are not components, but just views on the site manager, the directive is part of the browser namespace.
Line 2: This is the interface under which the utility is registered.
Line 3-4: Here we provide a human-readable title and description for the tool, which is used in the tools overview screen.

The second step is to create a nice “Overview” screen that tells us the available local and acquired smileys available for a particular theme. The first step is to create a view class, which provides one method for retrieving all locally defined smileys and one method that retrieves all acquired smileys from higher up themes. In a new file called browser.py add the following code:

1  from zope.app import zapi
2
3  from localtheme import queryNextTheme, getURL
4
5  class Overview(object):
6
7      def getLocalSmileys(self):
8          return [{'text': name, 'url': getURL(smiley, self.request)}
9                  for name, smiley in self.context.items()]
10
11      def getAcquiredSmileys(self):
12          theme = queryNextTheme(self.context, zapi.name(self.context))
13          map = theme.getSmileysMapping(self.request)
14          return [{'text': name, 'url': path} for name, path in map.items()
15                  if name not in self.context]

Line 7-9: Getting all the locally defined smileys is easy; simply get all the items from the container and convert the smiley object to a URL. The return object will be a list of dictionaries of the following form:
- “text” - This is the text representation of the smiley; in this case the name of the smiley object.
- “url” - This is the URL of the smiley as located in the theme. We already developed a function for getting the URL ( getURL()), so let’s reuse it.
Line 11-15: We know that getSmileysMapping() will get us all local and acquired smileys. But if we get the next theme first and then call the method, we will only get the acquired smileys with respect to this theme. We only need to make sure that we exclude smileys that are also defined locally. From the mapping, we then create the same output dictionary as in the previous function.

The template that will make use of the two view methods above could look something like this:

1  <html metal:use-macro="views/standard_macros/view">
2  <head>
3    <title metal:fill-slot="title"
4           i18n:translate="">Smiley Theme</title>
5  </head>
6  <body>
7  <div metal:fill-slot="body">
8
9    <h2 i18n:translate="">Local Smileys</h2>
10    <ul>
11      <li tal:repeat="smiley view/getLocalSmileys">
12      <b tal:content="smiley/text"/> →
13      <img src="" tal:attributes="src smiley/url"/>
14      </li>
15    </ul>
16
17    <h2 i18n:translate="">Acquired Smileys</h2>
18    <ul>
19      <li tal:repeat="smiley view/getAcquiredSmileys">
20      <b tal:content="smiley/text"/> →
21      <img src="" tal:attributes="src smiley/url"/>
22      </li>
23    </ul>
24
25  </div>
26  </body>
27  </html>

Place the above template in a new file called overview.pt. All that’s left now is to register the view using a simple browser:page directive.

1  <browser:page
2      name="overview.html"
3      menu="zmi_views" title="Overview"
4      for=".localtheme.SmileyTheme"
5      permission="zope.ManageServices"
6      class=".browser.Overview"
7      template="overview.pt" />

30.6 Step V: Working with the Local Smiley Theme

Let’s test the new local theme now by walking through the steps of creating a utility via the Web interface. This will help us understand the tests we will have to write at the end. First restart Zope 3 and log in with a user that is also a manager. Go to the contents view of the root folder and click on the “Manage Site” link just below the tabs. When the screen is loaded, click on the “Tools” tab and choose the “Smiley Themes” tool. You can now add a new theme by pressing the “Add” button. Once the new page appears, enter the name of the theme in the text field and press the “Add” button. You best choose a name for the theme that is already used as a global theme as well, like “plain”. This way we can test the acquisition of themes better. Once the browser is done loading the following page, you should be back in the smiley themes tool overview screen listing the “plain” theme, which is already registered as being “active”.

Figure 30.1:

An overview of all smiley themes.

To add a new smiley click on “plain”, which will bring you to the theme’s “Contents” view. Right beside the “Add” button you will see a text field. Enter the name “:-)” there and press “Add”. You now created a new smiley. Click on “:-)” to upload a new image. Choose an image in the “Data” row and press “Change”, which will upload the image. Repeat the procedure for the “:)” smiley. To see the contrast, you might want to upload smileys from the “yazoo” theme.

Once you are done, click on the “Overview” tab and you should see the two local and a bunch of acquired smileys, which are provided by the global “plain” smiley theme.

Figure 30.2:

An overview of all available smileys in this theme.

If you like you can now go to the message board and ensure that the local smiley definitions are now preferred over the global ones for the “plain” theme.

30.7 Step VI: Writing Tests

While we have a working system now, we still should write tests, so that we can figure out whether all aspects of the local smiley theme are working correctly. The truly interesting part about testing any local component is the setup; once you get this right, the tests are quickly written.

When testing local components one must basically bring up an entire bootstrap ZODB with folders and site managers. Luckily, there are some very nice utility functions that help with this tedious setup. They can be found in zope.app.tests.setup. Here are the functions that are commonly useful to the developer:

setUpAnnotations(): This function registers the attribute annotations adapter. This function is also useful for placeless setups.
setUpTraversal(): This function sets up a wide range of traversal-related adapters and views, including everything that is needed to traverse a path, get object’s parent path and traverse the “etc” namespace. The absolute_url view is also registered.
placefulSetUp(site=False): Like the placeless setup, this function registers all the interfaces and adapters required for doing anything useful. Included are annotations, dependency framework, traversal hooks and the registration machinery. If site is set to True, then a root folder with a ServiceManager (also known as site manager) inside will be created and the site manager is returned.
placefulTearDown(): Like the placeless equivalent, this function correctly shuts down the registries.
buildSampleFolderTree(): A sample folder tree is built to support multi-place settings, something that is important for testing acquisition of local components. The following structure is created:
createServiceManager(folder,setsite=False): Create a local service/site manager for this folder. Note that the function can be used for any object that implements ISite. If setsite is True, then the thread global site variable will be set to the new site as well.
addService(servicemanager,name,service,suffix=""): This function adds a service instance to the specified service manager and registers it. The service will be available as name and the instance will be stored as name+suffix.
addService(servicemanager,name,iface,utility,suffix=""): Her we register a utility providing the interface iface and having the name name to the specified service manager. The utility will be stored in the site management folder under the name name+suffix.

Now we are ready to look into writing our tests. Like for the global theme, I decided to write doc tests for the theme. Thus, add the following lines in tests/test_doc.py after line 41:

1 DocTestSuite('book.smileyutility.localtheme')

The following tests will all be added to the doc string or the SmileyTheme class. We begin with calling the placefulSetUp() function and setting up the folder tree.

1  >>> from zope.app.tests import setup
2  >>> from zope.app.utility.utility import LocalUtilityService
3  >>> site = setup.placefulSetUp()
4  >>> rootFolder = setup.buildSampleFolderTree()

Next we write a convenience function that let’s us quickly add a new smiley to a local theme.

1  Setup a simple function to add local smileys to a theme.
2
3  >>> import os
4  >>> import book.smileyutility
5  >>> def addSmiley(theme, text, filename):
6  ...     base_dir = os.path.dirname(book.smileyutility.__file__)
7  ...     filename = os.path.join(base_dir, filename)
8  ...     theme[text] = Smiley(open(filename, 'r'))

Now that the framework is all setup, we can add some smiley themes in various folders.

1  Create components in root folder
2
3  >>> site = setup.createServiceManager(rootFolder)
4  >>> utils = setup.addService(site, zapi.servicenames.Utilities,
5  ...                         LocalUtilityService())
6  >>> theme = setup.addUtility(site, 'plain', ISmileyTheme, SmileyTheme())
7  >>> addSmiley(theme, ':)',  'smileys/plain/smile.png')
8  >>> addSmiley(theme, ':(',  'smileys/plain/sad.png')
9
10  Create components in `folder1`
11
12  >>> site = setup.createServiceManager(rootFolder['folder1'])
13  >>> utils = setup.addService(site, zapi.servicenames.Utilities,
14  ...                          LocalUtilityService())
15  >>> theme = setup.addUtility(site, 'plain', ISmileyTheme, SmileyTheme())
16  >>> addSmiley(theme, ':)',  'smileys/plain/biggrin.png')
17  >>> addSmiley(theme, '8)',  'smileys/plain/cool.png')

Line 3: First, we make the root folder a site.
Line 4-5: There are no local services in a new site by default. Before we can add utilities, we first need to add a local utility service to the site.
Line 6-8: First we create the theme and add it as a utility to the site. Then just add two smileys to it.
Line 10-17: A setup similar to the root folder for folder1.

Now we have completely setup the system and can test the API methods. First, let’s test the getSmiley() and querySmiley() methods via the package’s API convenience functions.

1  Now test the single smiley accessor methods
2
3  >>> from zope.publisher.browser import TestRequest
4  >>> from zope.app.component.localservice import setSite
5  >>> from book.smileyutility import getSmiley, querySmiley
6
7  >>> setSite(rootFolder)
8  >>> getSmiley(':)', TestRequest(), 'plain')
9  'http://127.0.0.1/++etc++site/default/plain/%3A%29'
10  >>> getSmiley(':(', TestRequest(), 'plain')
11  'http://127.0.0.1/++etc++site/default/plain/%3A%28'
12  >>> getSmiley('8)', TestRequest(), 'plain')
13  Traceback (most recent call last):
14  ...
15  ComponentLookupError: 'Smiley not found.'
16  >>> querySmiley('8)', TestRequest(), 'plain', 'nothing')
17  'nothing'
18
19  >>> setSite(rootFolder['folder1'])
20  >>> getSmiley(':)', TestRequest(), 'plain')
21  'http://127.0.0.1/folder1/++etc++site/default/plain/%3A%29'
22  >>> getSmiley(':(', TestRequest(), 'plain')
23  'http://127.0.0.1/++etc++site/default/plain/%3A%28'
24  >>> getSmiley('8)', TestRequest(), 'plain')
25  'http://127.0.0.1/folder1/++etc++site/default/plain/8%29'
26  >>> getSmiley(':|', TestRequest(), 'plain')
27  Traceback (most recent call last):
28  ...
29  ComponentLookupError: 'Smiley not found.'
30  >>> querySmiley(':|', TestRequest(), 'plain', 'nothing')
31  'nothing'

Line 7: Set the current site to the root folder. All requests are now with respect from that site.
Line 8-11: Make sure that the basic local access works. Note that the TestRequest defines the computers IP address to be 127.0.0.1 and is not computer-specific.
Line 12-17: Make sure that a ComponentLookupError is raised, if a smiley is not found or the default is returned, if querySmiley() was used.
Line 19-31: Repeat the tests for using folder1 as location. Specifically interesting is line 22-23, since the smiley is not found locally, but retrieved from the root folder’s theme.

Let’s now test the ‘getSmileysMapping()‘ method. To do that we create a small helper method that helps us compare dictionaries.

1  >>> from pprint import pprint
2  >>> from book.smileyutility import getSmileysMapping
3  >>> def output(dict):
4  ...     items = dict.items()
5  ...     items.sort()
6  ...     pprint(items)
7
8  >>> setSite(rootFolder)
9  >>> output(getSmileysMapping(TestRequest(), 'plain'))
10  [(u':(', 'http://127.0.0.1/++etc++site/default/plain/%3A%28'),
11   (u':)', 'http://127.0.0.1/++etc++site/default/plain/%3A%29')]
12
13  >>> setSite(rootFolder['folder1'])
14  >>> output(getSmileysMapping(TestRequest(), 'plain'))
15  [(u'8)', 'http://127.0.0.1/folder1/++etc++site/default/plain/8%29'),
16   (u':(', 'http://127.0.0.1/++etc++site/default/plain/%3A%28'),
17   (u':)', 'http://127.0.0.1/folder1/++etc++site/default/plain/%3A%29')]
18  >>> getSmileysMapping(TestRequest(), 'foobar')
19  Traceback (most recent call last):
20  ...
21  ComponentLookupError: \
22  (<InterfaceClass book.smileyutility.interfaces.ISmileyTheme>, 'foobar')

Line 8-17: Again, test the method for two locations, so that acquisition can be tested.
Line 18-22: Make sure we do not accidently find any non-existent themes.

After all the tests are complete, we need to cleanly shutdown the test case.

1 >>> setup.placefulTearDown()

You should now run the tests and see that they all pass. Another interesting function that deserves careful testing is the queryNextTheme(). I will not explain the test here, since it is very similar to the previous one and will ask you to look in the code yourself for the test or even try to develop it yourself.

Exercises

Something that I have silently ignored is to allow to specify a default smiley theme. This can be simply accomplished by adding a second registration for a theme. Implement this feature.
Currently, smileys are always acquired. But this might be sometimes undesired and should be really up to the manager to decide. Develop an option that allows the manager to choose whether smileys should be acquired or not.
Uploading one smiley at a time might be extremely tedious. Instead, it should be allowed to upload ZIP-archives that contain the smileys. Implement that feature.

Chapter 30Local Utilities