ZEO cache

Why you should care about the ZEO client cache

Fetching an object from ZEO over the network can be MUCH slower than fetching the object from a local FileStorage, depending on the network of course, but even on localhost ZEO has some overhead that can be significant -- especially for loading Images and Files. If instead the object is in the ZEO client cache, no network request is made and the speed is basically equivalent to FileStorage.

In the common case, when you have a lot of objects that are read more often than they are written, a big ZEO client cache can be very helpful.

Tuning the Cache Size

Jim Fulton says: I would definitely err in the direction of using a larger cache.

As of today (Zope 2.9.2), the ZEO client cache is still 20 MB by default, which dates back many years and is now much too small for most "real" storage sizes.

To tune your cache, you can enable cache trace logging. This is documented in ZODB's doc/ZEO/trace.txt. Also read cache.txt in the same directory which gives a brief overview of the ZEO cache design.

In Zope 2.7 and earlier, there was an easier clue that you needed to increase the cache size. See how often there are lines like this in your event log:

2006-01-25T10:30:59 INFO(0) ZEC:1-None-0 flipping cache files. new current = 0

There is no hard rule on how many flips is "too many"; one suggestion is that you don't want flips more often than ten minutes, under load.

In Zope 2.8 and later, the cache is one big file and there are no more "flip" messages, so you must use the trace tools described above.

Tim Peters commenting on the performance impact of cache tracing:

It's definitely intended that you be able to use ZEO's cache tracing on a production box.

No particular memory burden (trace records are written directly to a disk file, not in any way cached in memory (apart from that the OS may use RAM to buffer disk writes, of course).

The speed burden should be minor: producing a trace record requires a trivial amount of computation, and then whatever time it takes to pass a binary string of a few dozen bytes off to the platform output routines (note that this is summary info, and trace records have a fixed small size independent of object sizes). Given that a ZEO cache hit has to do hundreds or thousands or ... bytes worth of file I/O anyway (to read up the object pickle), and a cache hit is the cheapest thing you can do with ZEO, it's relatively minor additional work.

Depending on cache activity, trace files can grow quickly (megabytes per hour is common, and I contrived tests that produced hundreds of megabytes per hour), and that's probably the biggest thing to look out for. For example, if the trace file is on a small partition, don't be suprised if a traced ZEO craps out with an "out of disk space" error when trying to append a new trace record to the file.

Issues with Persistent Client Caches

Tim Peters, about persistent client caches: OTOH, the larger the (persistent) ZEO cache file, the longer it may take (zope) startup or reconnection cache verification to complete, so there's always some reason not to do the obvious thing <0.3 wink>.

Jim Fulton notes: We seem to have a lot of problems with persistent caches for some reason, so I tend to recommend against their use. I'm not sure what's going on there. I still find them useful in situations in which the connection to the (ZEO) server is slow, ... (but that) case should be rare.

Tim replied: I believe there are multiple causes, from undetected cache-file corruption after a non-clean shutdown (...although I tried to add a little more sanity-checking in the post-MVCC cache), to users switching the database their ZEO client connects to without deleting the persistent cache file(s) or changing any of the stuff on the client that gets folded into that file's name. The key seems to be that almost all cases of persisent-cache problems I'm aware of "got fixed by magic" just by deleting the cache file(s) and restarting.

Pascal Peregrina notes: About the whole cache being verified every time an instance is restarted: This might indicate that the ZEO invalidation queue size is too small. So something useful to test is to increase the invalidation-queue-size param in zeo.conf (<zeo></zeo> section). I think default size is 100, and we solved our Zope slow startup issues by increasing to 1000. Note that the ZEO instance will consume more memory to keep this information.