Blobs Zero Copy

Consuming blob files

I propose to extend the interface IBlob? with a new method consumeFile:

def consumeFile(filename):
    """Will replace the current data of the blob with the
    file given under filename.

    This method uses link-like semantics internally and has the
    requirement that the file that is to be consumed lives on the
    same volume (or mount/share) as the blob directory.

    The blob must not be opened for reading or writing when
    consuming a file.
    """

Using this method will allow the consumption of large files into a Blob with a runtime of O(1) as the link operation is O(1).

Fallback not: Link-like operations will fail if the source is on a different volume than the destination. We could allow falling back to the current behaviour of copying the data, although this is very inefficient.

Note: To use this method in Zope 2 and Zope 3, both publishers have to start using a temporary file approach that allows access to the filename. Fortunately in both cases a small tweak using a subclassed cgi.FieldStorage? to use tempfile.NamedTemporaryFile? is sufficient. However, the change of Zope is out of scope for this proposal.

Transferring blob files to ZEO

To avoid copying files over the ZEO protocol I propose to allow the use of a shared writable filesystem. The Blob storage area implementation already allows the use of a shared readable filesystem. In addition, as the filenames that get created for temporary uncommitted data are unique among all ZEO clients, those can be stored in the shared filesystem as well as no collisions are expected.

A risk of collision is that two ZEO clients would start committing two transactions with the same tid in parallel. This can not happen, as the tid is computed on the ZEO server for the ZEO client when running tpc_begin. The commit lock is already acquired by the ZEO client at this moment, so this race condition is safe.

The implementation will introduce a new option for the ClientStorage? named blob-cache-writable which will default to False. When this option is set to False the current implemented behaviour of transmitting the file over the ZEO protocol will be used.

When setting the option to True the ClientStorage? will use a new zrpc method storeBlobShared only transmits the relative filename of the blob. The implementation of storeBlobShared will use the argument filename to transmit the filename (relative to the blob`s oid path) to the server.