home contents changes options help subscribe

last edited 9 years ago by Brian

Web Services Goals

This document provides a high-level overview of what we are trying to accomplish in supporting Web Services in Zope 3, as well as establishing some scope for the project.

Overview

Web Services are packages of functionality that are published to the network and can be used as building blocks for other software. They can be thought of in many ways like a network-wide, cross platform "component model".

The IBM developerWorks Web Services Site contains a number of good resources and is a good starting point for learning more about web services. In particular, the article The Web Services Revolution is a good introduction to web services and the benefits (and challenges) of the web services model.

There is currently some evidence (such as .NET) that Web Services could become an important development model. Currently, Zope applications are not poised to take advantage of this development model as the infrastructure provides no support for core Web Services technologies such as SOAP and WSDL.

Web Services != SOAP

It is important to note that "Web Services" is more than SOAP. To be a viable network component system, Web Services need to do more than just speak to each other in a common protocol. They need to be managable from a deployment perspective. They need to be discoverable and provide descriptions of themselves to interested clients. They need to interact in well-defined ways with their deployment environment (transactions, etc.).

In other words, SOAP is only the TCP/IP of Web Services; a successful Web Services platform needs to support the larger concepts in order to be useful beyond the "novelty stage" of SOAP.

By "novelty stage" I mean that right now because SOAP is new and there is a general lack of tools, developers are willing to write pretty low-level SOAP code. In the not-too-distant future, most developers will have tools that take care of all of those details. Once those tools become entrenched, developers will no more write SOAP-level code to access a Web Service than a developer would write TCP/IP level code today in a function to grab a Web page.

We need to take this fact into account when deciding how to provide Web Services support in Zope. There are a number of (Python) SOAP tools available now, but none are yet comprehensive enough to provide a long-term platform.

Note that this is not meant as a criticism of the tools currently available - given the rate of change of WS technologies and the uncertain adoption curve, the approaches taken so far are perfectly reasonable.

I yammer on about this to be sure that we frame the problem statement correctly for this project. The problem statement is "Zope lacks the infrastructure support required to implement and consume Web Services". It is not (I assert) that "Zope lacks SOAP support" (see below).

Why not just SOAP?

There are definite uses for SOAP that do not carry all of the rest of the Web Services baggage. For some Zope users, it would be useful to have SOAP support similar to the xml-rpc support we have now in Zope 2 (for example, for supporting a thick client that uses SOAP for RPC to a Zope process).

However, there are many people who are uncomfortable with that kind of blanket support. While it is expedient from the point of view of a developer, it poses a number of manageability problems that are legitimate. If people are not "comfortable" with Zope, they won't use it, and we are trying to be more cognizant of this.

So what does that mean for those who want blanket SOAP support in Zope? It does not mean you are out of luck. It just means that for this first project we are concentrating on "Web Services" because:

  • They are less controversial from a comfort and management perspective
  • They are closer to the conceptual model that people in the WS world already have
  • Implementing them will produce the key infrastructure to make other kinds of SOAP support easier

One should not read into this proposal that Zope 3 will never support "magic" SOAP support being implemented for those who want it. That is just not one of the goals of this particular project, and we think that this project serves as a more bounded first step.

Proposed Architecture

A robust Web Services infrastructure for Zope would consist of a number of conceptual "layers". These layers provide some protection against the rate of change of the underlying technologies, and (more importantly) provide the flexibility that will be required as "Web Services" as a programming model evolves and matures.

Moreover, most of those layers should be Zope-independent (usable by any Python programmer as a normal Python package). This ensures that the "Zope layer" will be relatively thin, and that it will be relatively easy for people to build an alternative or extended Zope layer for special needs. This pattern is repeated at succeeding lower layers, for example: 

      ----------------------------------------------------------------------
      |   Zope layer       (Drop-in objects to build a Web Service and to  |
      |                     represent a proxy to a remote service)         |
      ----------------------------------------------------------------------

      ----------------------------------------------------------------------
      |   High-level layer (objects representing services and proxies      |
      |                     to remote services, high-level utilities)      |
      ----------------------------------------------------------------------

      ----------------------------------------------------------------------
      |   Abstract layer   ("conceptual" level objects like SOAP messages, |
      |                     service descriptions, etc. that expose enough  |
      |                     to do interesting things w/o requiring extreme |
      |                     carnal knowledge of low-level details)         |
      ----------------------------------------------------------------------

      ----------------------------------------------------------------------
      |   Low-level layer  (encapsulates the gory details of parsing xml   |
      |                     descriptions, messages, transport, encoding,   |
      |                     etc.)                                          |
      ----------------------------------------------------------------------

The high-level (Python) layer will provide objects such as service proxies that automate the details of calling remote services and provide all that is needed for the most common usage. Because the high-level layer is thin, it will be easy to implement alternative high-level objects that deal with more complex situations (for example, transaction integration or special types of security).

The "abstract" layer models the domain abstractions having to do with Web Services (messages, metadata, service descriptions, etc.) without forcing the programmer to worry about too many low-level details.

The low-level layer handles the low-level details like xml parsing, and production, message encoding and transport.

Because most of the flux in the various specs related to Web Services is in the details, this architecture will provide some insulation to application developers. As the specs evolve, much of the changes can be taken care of in the lower layers of the architecture.

Most interesting for this proposal is what the Zope 3 layer of this architecture would look like.

Need to fill this in, maybe after some IRC or sprint time.

Some general notes on the Zope object layer:

Scalability
The primary goal of the Zope-level objects in this implementation is ease-of-use and fast development, not raw (execution) speed. The performance goal for this project is to allow people to quickly and easily define and implement Web Services via the Web that are adequate for local-network (or "departmental") types of usage. The goal is specifically not to provide a way to implement extremely high-volume public services like a Passport in Zope.

In fact, a big part of the reason for partitioning the architecture as I have is so that it would be possible to implement a "lean and mean" high-volume web service (with or without Zope) without having to start from scratch. Implementing such services are beyond the scope of this proposal.

Non-Magic
I have intentionally avoided the "magic" approach in thinking about the Zope manifestation of Web Services. The "magic" approach is the one taken by the current Zope 2 xml-rpc support - that everything in Zope is automagically callable via xml-rpc.

While that works OK for xml-rpc, the idea of a "Web Service" is much more about a specifically bounded package of functionality with a known identity and certain self-descriptive capabilities. While it would be technically possible to support SOAP as we support xml-rpc, in the long term that would provide little value over the current xml-rpc support.

Clients of web services have certain expectations, and the magic approach does not fit with either the mindset or the tools approach that the "Web Services community" is adopting. The idea of a Web Service as a well-bounded component with certain managability, security and deployment requirements is also well-entrenched in the WS community.

Finally, the fact that SOAP supports far more complex constructs than xml-rpc coupled with Python's highly dynamic typing make it very difficult to do "magic" that works reliably for more than the most simplistic cases. To do useful things, you need to get hints from a human, which the current proposal does. (There, have I argued against magic enough? :)

That said, the architecture should make it relatively easy to provide some "magic" exposure of methods via SOAP if one were inclined to do so as a third-party product. I do not expect such magic to become part of core Zope 3 anytime soon though.

Optimization
Particularly for Zope-based service proxies, there are some opportunities for optimization that we could consider. The biggest is caching. We might want to think about providing some per-method knobs that would allow a site manager to tune a service proxy to scale better.

For example, a remote service like a "stock quote service" could often be optimized by arranging for it's getQuote method to cache results (based on argument values) for certain intervals.

Risk Factors

  • Rate of change. Web Services is an umbrella term for a number of related technologies (SOAP, WSDL, UDDI and other acronyms). Each of these technologies is still in a fairly rapid state of change, which imposes a certain amount of risk for this project.

    That said, a fair amount of infrastructure has now been released by a large Redmond company in the form of .NET, with a big marketing blitz and a lot riding on adoption. Other large companies are also readying Web Services products, so these rollouts should have a de facto stabilizing effect on the rate of change.

    The rate of change of the technologies has also slowed enough on its own that we can, to a large extent, mitigate this risk by designing with the fact of change in mind. Over time it will be necessary to support multiple versions of these different technologies anyway, so we can minimize the impact by keeping the design loosely coupled and providing clear extension points up front.

  • Metadata. Because Python is not strongly-typed, it can be difficult to discover some of the kinds of information that you would otherwise get "for free" in a strongly-typed language.

    The WS world is pretty interface-centric and the interfaces tend to be strongly-typed. This might be a problem re: Zope 3 "interfaces" which provide signatures and documentation but not necessarily any type information.

Scope

The scope of this project is to design and implement a solid Python infrastructure for building and using Web Services and to provide "built-in" Zope objects that allow Web Services to be created and used in the Zope "through-the-web" environment. The through-the-web objects should focus on relatively simple common usage, and be based on an underlying infrastructure that provides the flexibility to support more advanced usage.

We are specifically not going to add an "everything-is-automagically- SOAP-callable" capability to the Zope 3 core. The infrastructure should be robust enough that the community could easily develop that kind of functionality as an add-on if people really want it.

To a large extent, the "discovery" aspect of Web Services (UDDI) is out of scope of this proposal. I think that it will be possible to add this later without impacting the architecture. For example, we could choose later to provide a "Service Proxy Wizard" that uses UDDI to let you select a service to proxy via a Web form.

Also, support for XML-RPC, REST or other similar web-service-ish technologies are not part of the goals for this project.

References