Running asynchronous code

Status

Author

Tarek ZiadÃÂ© - tziade@nuxeo.com

Problem/Proposal

A web application has to deal sometimes with third party pieces of softwares, like an IMAP or SMTP server, a FTP Server or some other things on the system.

Calling theses third party actors in Zope 3 is always done within a process of rendering a page.

Sometimes, part of the process can be done without having to wait for these actors to finish their work, so the system can get back to the user.

There are many examples of this need, and the most obvious one has been already done in zope.app.mail package :

A threaded process queue that can be used through an utility to send emails in a separate thread via SMTP.

I would like to propose a global mechanism that would allow to run asynchronous pieces of code.

This asyncrhonous idea was already mentioned is an existing proposal :

http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/TaskSchedule

But this proposal is way more sophisticated, as it is more likely to be cron jobs for Zope.

Furthermore, I would like to present a different approach on the asyncrhonous problem and this would maybe become a part of TaskSchedule proposal later.

Goals

Create a global utility that can be called by a Zope 3 application to execute pieces of code in a separate thread
Provide this mechanism for all sessions
Provide a way to lock parts of ZODB

Proposed Solution

This solution was inspired by the zope.app.mail threaded queued

A global utility is provided and the Zope 3 application can feed a function wrapped into an IASyncFunction? adapter.
Everytime a function is provided, the utility looks for the current session id and looks if it has a thread dedicated to this particular id.
If no thread exists for this session id, it creates the thread and holds it.
Then, the global utility feeds the thread with the given function.
The method is executed by a thread and an event is triggered giving feedback :
+ the function succeeds, a IASyncFunctionDone? is dispatched and might contain function results

+ the function raises an error, a IASyncFunctionRaised? is dispatched containg error infos
The thread dies when it has finished its works. This means that the thread can receive other work to do while it's working.

Location

the location could be in a new package in zope.app, called async

Risks

Conflict Error and Transaction Handling issues may raise in such a multi-threaded environment need to find a efficient way to lock objects used by the thread
On the contrary of zope.app.mail, if Zope crashes, the queue is lost as we can hardly save the jobs in such a generic utility.

comments:

How is handling the scheduling info for local components? --rogerineichen, 2005/04/02 06:47 EST reply
That's a good idea. I need such a mechnism for execute our pop3 mailinglist client with something like a background process. How do we interact with this utility? I think my usecase could be. Somebody can add a (mailinglist) utility and register this utility with schedule information. At least one component has to be local and store persistent information about the scheduling info.