This proposal has been implemented.
The current implementation of blobs requires to copy data from blob files at two points:
- Pulling a blob into the system
When pulling a blob initially into the system, we need to do something like:>>> blob = Blob() >>> writing = blob.open('w') >>> ... # copy over blob data from original file chunkwise >>> writing.close()
This is not efficient for handling large files. Because it has an O(N) runtime. (N being the size of the file)
- When using ZEO the blob will be transferred chunk-wise to the ZEO server while actually holding the commit lock and so blocking other transactions to finish. Again, this has a runtime behaviour of O(N).
I propose to extend the interface IBlob? with a new method consumeFile:
def consumeFile(filename): """Will replace the current data of the blob with the file given under filename. This method uses link-like semantics internally and has the requirement that the file that is to be consumed lives on the same volume (or mount/share) as the blob directory. The blob must not be opened for reading or writing when consuming a file. """
Using this method will allow the consumption of large files into a Blob with a runtime of O(1) as the link operation is O(1).
Fallback not: Link-like operations will fail if the source is on a different volume than the destination. We could allow falling back to the current behaviour of copying the data, although this is very inefficient.
Note: To use this method in Zope 2 and Zope 3, both publishers have to start using a temporary file approach that allows access to the filename. Fortunately in both cases a small tweak using a subclassed cgi.FieldStorage? to use tempfile.NamedTemporaryFile? is sufficient. However, the change of Zope is out of scope for this proposal.
To avoid copying files over the ZEO protocol I propose to allow the use of a shared writable filesystem. The Blob storage area implementation already allows the use of a shared readable filesystem. In addition, as the filenames that get created for temporary uncommitted data are unique among all ZEO clients, those can be stored in the shared filesystem as well as no collisions are expected.
A risk of collision is that two ZEO clients would start committing two transactions with the same tid in parallel. This can not happen, as the tid is computed on the ZEO server for the ZEO client when running tpc_begin. The commit lock is already acquired by the ZEO client at this moment, so this race condition is safe.
The implementation will introduce a new option for the ClientStorage? named blob-cache-writable which will default to False. When this option is set to False the current implemented behaviour of transmitting the file over the ZEO protocol will be used.
When setting the option to True the ClientStorage? will use a new zrpc method storeBlobShared only transmits the relative filename of the blob. The implementation of storeBlobShared will use the argument filename to transmit the filename (relative to the blob`s oid path) to the server.
The implementation of both proposed changes will allow the Blob support in ZODB to not require any O(N) operations when committing Blobs into the ZODB if the data is available as a file on the shared writable storage initially.
I'd like to get feedback on:
- whether or not to fall back when link() doesn't work for consuming temporary file as a blob
- As pointed out, the Zope publishers should be adapted to provide a temporary file based on tempfile.NamedTemporaryFile? to allow application code to call consumeFile.
- The Zope publishers also should implement early processing of the request body to avoid blocking the application thread while processing an uploaded multi-part request. (Processing of a multipart body with an embedded 50MB file takes about 5 seconds on my machine.)
FWIW, Windows supports Hard Links --sidnei, Thu, 08 Mar 2007 04:27:05 +0000 reply
The omission on os.link() is just the lack of a good soul to contribute it apparently ?. A good task for a friday evening.
? http://msdn2.microsoft.com/en-us/library/aa363860.aspx ? http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/fsutil_hardlink.mspx?mfr=true ? http://www.flexhex.com/docs/articles/hard-links.phtml ? http://www.thescripts.com/forum/thread537011.html
And if you need to check if two 'files' point to the same... --sidnei, Thu, 08 Mar 2007 04:31:16 +0000 reply