home contents changes options help subscribe edit (external edit)

last edited 3 years ago by tim_one

ZEO 2 Proposal

Current status

The proposal is being developed.

An implementation of Phase 1 is mostly complete and an alpha release will be available in mid- to late-April. The development is being done on the ZEO-ZRPC-Dev branch of ZEO.

The initial release has been delayed because of a current emphasis on unit tests for ZODB storages. The tests have uncovered several minor bugs in ZEO 1 and ZEO 2. There will be a ZEO 1.0 beta 2 release in early May, followed shortly by a ZEO 2 alpha release.

Problem Statement

The client-server protocol used by ZEO needs to be improved. The existing protocol has limitations that hamper future development of ZEO and reuse of the protocols for related projects like Replicated Storage.

There are several layers of communication software used by ZEO. At the lowest level, ZEO uses the asyncore library for asynchronous socket communication. It uses asyncore to build a higher level, RPC-style library implemented in the zrpc module. Finally, ZEO defines a client-server storage protocol using zrpc.

The ZEO 2 project focuses primarily on the middle layer. It will provide a new RPC protocol that improves on the current protocol in several ways.

  • The old protocol is asymmetric. A client invokes methods on the server, but the server sends messages back to the client using the "out-of-band" hook. The interface between the RPC layer and the application layer is different for each style of communication.
  • The implementation of zrpc must accomodate several styles of asyncore use. When used with a multi-threaded application like Zope, the thread that invokes zrpc communication is not the thread that invokes the asyncore mainloop. The implementation of this support in zrpc is complicated.
  • The zrpc protocol module does not implement full support for generic invocation of remote methods; it defers some of the communication to the object that provides application logic. The new protocol will support method invocation of unmodified objects.
  • A client cannot make concurrent synchronous calls to a server. This limits flexibility and performance. The new RPC protocol should allow multiple calls to be outstanding. This change may require modifications to application protocols, which may depend on the single call behavior. (If a client receives an asynchronous message from the server, it may assume the message applies to the transaction that is currently making a synchronous call.)
  • The protocol does not include a means for graceful evolution, like a protocol version number.

There are several other improvements to ZEO that will be considered during the ZEO 2 project, although not necessarily during the first phase of implementation.

  • Multiple transaction commits in parallel. The ZEO storage server only allows a single client transaction to commit at one time. This limits performance, because all the I/O for a transaction occurs after the commit begins. The new server implementation should allow multiple transactions to commit in parallel; only the tpc_vote() call should require locking.
  • Improved cache validation performance. The implementation of the ZEO client cache could be improved. The current cache reads the cache file twice, once to open the cache and another to perform validation. The replacement policy and storage management could also be improved.

Architecture

  • XXX Not here yet
  • Note on multiple transaction commits: Extra copying. The StorageServer? would need to buffer stores until tpc_vote() is called, then it would actually perform the transaction on the storage. But the storage itself will also buffer the stores until it finds out if the transaction commits. As a result, we make two copies of all the transaction data in two different files.

Risk Factors

  • Backwards compatibility.

    The new RPC protocol will not be backwards compatible with the old RPC protocol. Users of earlier releases, up to and including ZEO 1.0, would need to upgrade their installations to use ZEO 2.

    This is a problem for users in a production system; a flag day for the new install could be disruptive. Instead, we plan to provide an adapter that allows pre-2.0 clients to talk to a 2.0 server. This approach would allow users to upgrade servers individually. When all the servers are running 2.0, clients can be upgraded individually.

Deliverables & Schedule

There will be two phases to the implementation, with a more modest set of deliverables for phase one.

  • Phase 1

    The first phase of the project will focus on the lower protocol layers. Its goal is to make the new architecture available as soon as possible. Other projects, such as Replicated Storage, cannot being until this phase is completed.

    XXX Explicit list of deliverables

  • Phase 2

    The second phase of the project will focus on application layer improvements and backwards compatibility.

    XXX Explicit list of deliverables

jim (Apr 18, 2001 7:45 am; Comment #1)
 >     - A client cannot make concurrent synchronous calls to a server.
 >       This limits flexibility and performance.  The new RPC protocol
 >       should allow multiple calls to be outstanding.  This change may
 >       require modifications to application protocols, which may depend
 >       on the single call behavior.

I don't think that this is right.

 >  (If a client receives an
 >       asynchronous message from the server, it may assume the message
 >       applies to the transaction that is currently making a
 >       synchronous call.)

This is not an rpc issue. It is an application level issue. If multiple transactions can be active, then asynchronous messages relating to a transaction must be labelled.

jeremy (Apr 18, 2001 10:39 am; Comment #2)
You are correct that this is an application-level issue. I was trying to say that the application currently depends on the rpc layer's feature that there is only a single outstanding synchronous call and thus a single transaction. If the lower layer allows multiple synchronous calls, the application protocols must change in subtle ways to take advantage of this feature, i.e. the asynchronous replies must indicate the transaction.
bavarian (May 1, 2001 2:36 pm; Comment #3) Editor Remark Requested
Are there any plans to develop something like "SecureZEO?"? I don't mean securing a server-to-server ZEO connection, which can easily be handled on the transport layer via stunnel or so. It's about using ZEO with non-Zope clients, e.g. for editing (wxPython to ZODB) or as a synchronization protocol with offline client-side Zope instances (we are testing this for a groupware project). The problem to be solved is that ZEO currently does not enforce Zope security (roles, users, permissions), and that the clients we are thinking of are not "trusted" like in the distributed Zope servers scenario of zope.org.
  • There's been a fairly length discussion of this issue on the zodb-dev mailing list. The short answer is the untrusted clients can't use the ZEO protocol because it gives them access to object pickles. Instead, you'd need something like a trusted ORB that served objects to untrusted clients via RPC. --jeremy
jeremy (May 2, 2001 9:56 am; Comment #4)
 >  Are there any plans to develop something like "SecureZEO"?

We have talked about doing this, but it isn't a top priority currently. I expect we would get to it early fall unless lots of users clamor for it before then.

smog (Feb 28, 2002 8:42 pm; Comment #5)
re: "SecureZeo?"

I would really like to see at least IP control and whether a client can read/write or just read from the ZODB. I have hacked the current ZEO 1.0x stuff. I would really like to see at least this much in the base ZEO implementation.


comments:

Replication --davidsumner, 2003/10/27 16:38 EST reply
What approach was taken to introduce replication by zope.com in ZEO? Have they used unicast or multicast? Has anyone looked at www.spread.org as a possible transport for replication purposes? There is a python interface to spread.



subject:
  ( 11 subscribers )