PersistentCodeImplementationPlan

Goals:

• use ZODB 3.3 in Zope 3
• fix Zope 2.8 so ZClasses work
• make ZODB 3.3 deal with persistent references in the manner of ZODB 4; this is a major change in the way ZODB/Zope2 handles these references. References to ZClasses (Zope 2) and Persistent Classes (Zope 3) need to work in the same way.
• broken object support (from Zope 2) needs to be kept (abstractly); we'll need a new implementation of broken objects since the existing impl can't handle the new approach to persistent class references

Implementation notes

Broken-object approach

• classes from file-based modules:
  • replace class loader for the pickler to provide a replacement class when normal class loading fails
    • enhance cPickle with a hook that is called only when the normal class loader fails (Python 2.4, custom cPickle for Python 2.3.x)
    • change export/import to avoid copying classes referenced from their instances (via __class__)
• for ZClasses / Persistent Classes:
  • when we delete a class, it needs to become some sort of Broken Object implementation so that instances show up as broken objects.

    we can probably insert a new base class, but probably need to clear the class __dict__ as well; using __getattribute__ in the base class can avoid allowing the persistent class to interfere with many operations of the broken behavior, but type slots filled for __*__ methods remain an issue

• for deep copy, we'll need a custom pickler to avoid copying references to __class__ like any other reference (this avoids copying the class itself into the instance hierarchy)

Milestones:

1. Start
   • send message to zope-dev and zope-coders discussing the plan and the issue of breaking the broken object support; note that ZClasses are already seriously hosed on the Zope 2 head

     Status: Completed

2. Zope 3 working with ZODB 3.3
   • change persistent instance pickling to save a direct reference to the __class__
     • loading old-style references must continue to work (to deal with existing databases)
       • need to create and capture good examples to test with

     Status: Completed

   • change pickling to use __getnewargs__ instead of __getinitargs__() (which is never used for new-style objects; all persistent instances are new-style objects)

     Status: Completed

   • add "transaction" package/module facade for "ZODB.Transaction", no warnings since Jim wants to make this the new API for this

     Status: Completed

   • make old imports work with warnings: "persistance"

     Status: Completed

   • continue special casing in the object cache to avoid deactivating persistent classes (test: isinstance(obj, type)); all other special casing will be strenuously avoided.

     Status: Completed

   • move zodb.code support over from ZODB 4; needs review. Is known to have problems related to invalidation. (class docstrings are being lost)

     Note that for now, we moved this to zodbcode.

     Status: Completed

   • metatype for persistent classes needs to override the descriptor for __doc__ so the attribute can be mutated as the source changes
   • backport weak reference support (Jim's persistent weak references)

     Note: still in persistence package, still needs to be moved to either persistent or ZODB.

     Status: Completed

   • backport early object id allocation patch from Steve

     What's the status of this?

3. Zope 2.8 broken objects
   • create a custom class loader for the unpickler:
     • use the existing ZODB API to set the class loader
     • new approach: call the original loader, and return a broken object type when that fails
   • need UI to allow inspection of the broken objects (review existing implementation; check that it's sufficient)
   • need to be able to "break" a ZClass when deleted by making it behave as a broken object class before actually deleting it from it's container
4. Zope 2.8 ZClasses (currently broken anyway)
   • fix existing breakage
   • write test cases so it doesn't happen again!
5. Zope 2.8 export/import, copy/paste working
   • update to avoid copying __class__ references (this includes fixing the existing (if ZClasses weren't broken) bug that ZClasses get copied when instances are copied
   • need to ask if anyone outside Zope uses the export/import feature of ZODB (zodb-dev); if not, we just need a new copy/paste for Zope 2.8
6. Zope 3 Broken objects
   • create a broken object that can be used when classes can't be loaded

     Status: Completed

   • make persistent classes that have been removed from their modules (or if the module is deleted) become "broken object" implementations, similar to the Zope 2.8 flavor for ZClasses (milestone 2)

     Status: Deferred until after X3.0

Future tasks, not related to these milestones:

• make "ZODB.Transaction" a facade (with warning) for "transaction"

Expected Breakages:

• import/export, copy/paste for persistent classes (Zope 2.8; already broken)
• broken objects (Zope 2.8)

Terms:

invalidation

Some other thread commits change to and object; other copies need to throw away their state so they get reloaded when their state is needed:

         del o._p_changed     # classic syntax

Persistent classes should immediately reload state instead of relying on a delayed reload.

deactivation

Cache decides object hasn't been used in a while; we want to throw away the state to save resources (memory):

         o._p_changed = None  # classic syntax
         o._p_deactivate()    # alternate syntax

Persistent classes should ignore deactivation. Other objects are ghostified.

Current machinery assumes that o._p_state == GHOST after requesting deactivation, but persistent classes don't behave that way.

Currently classes never get in the ring of ghostifiable objects. Since persistent classes are never ghostifiable, they should never be added to the ring.