SF bug 839548: Bug in type's GC handling causes segfaults.
Also SF patch 843455.
This is a critical bugfix.
I'll backport to 2.3 maint, but not beyond that. The bugs this fixes
have been there since weakrefs were introduced.
diff --git a/Modules/gc_weakref.txt b/Modules/gc_weakref.txt
new file mode 100644
index 0000000..b07903b
--- /dev/null
+++ b/Modules/gc_weakref.txt
@@ -0,0 +1,107 @@
+Before 2.3.3, Python's cyclic gc didn't pay any attention to weakrefs.
+Segfaults in Zope3 resulted.
+
+weakrefs in Python are designed to, at worst, let *other* objects learn
+that a given object has died, via a callback function. The weakly
+referenced object itself is not passed to the callback, and the presumption
+is that the weakly referenced object is unreachable trash at the time the
+callback is invoked.
+
+That's usually true, but not always. Suppose a weakly referenced object
+becomes part of a clump of cyclic trash. When enough cycles are broken by
+cyclic gc that the object is reclaimed, the callback is invoked. If it's
+possible for the callback to get at objects in the cycle(s), then it may be
+possible for those objects to access (via strong references in the cycle)
+the weakly referenced object being torn down, or other objects in the cycle
+that have already suffered a tp_clear() call. There's no guarantee that an
+object is in a sane state after tp_clear(). Bad things (including
+segfaults) can happen right then, during the callback's execution, or can
+happen at any later time if the callback manages to resurrect an insane
+object.
+
+Note that if it's possible for the callback to get at objects in the trash
+cycles, it must also be the case that the callback itself is part of the
+trash cycles. Else the callback would have acted as an external root to
+the current collection, and nothing reachable from it would be in cyclic
+trash either.
+
+More, if the callback itself is in cyclic trash, then the weakref to which
+the callback is attached must also be trash, and for the same kind of
+reason: if the weakref acted as an external root, then the callback could
+not have been cyclic trash.
+
+So a problem here requires that a weakref, that weakref's callback, and the
+weakly referenced object, all be in cyclic trash at the same time. This
+isn't easy to stumble into by accident while Python is running, and, indeed,
+it took quite a while to dream up failing test cases. Zope3 saw segfaults
+during shutdown, during the second call of gc in Py_Finalize, after most
+modules had been torn down. That creates many trash cycles (esp. those
+involving new-style classes), making the problem much more likely. Once you
+know what's required to provoke the problem, though, it's easy to create
+tests that segfault before shutdown.
+
+In 2.3.3, before breaking cycles, we first clear all the weakrefs with
+callbacks in cyclic trash. Since the weakrefs *are* trash, and there's no
+defined-- or even predictable --order in which tp_clear() gets called on
+cyclic trash, it's defensible to first clear weakrefs with callbacks. It's
+a feature of Python's weakrefs too that when a weakref goes away, the
+callback (if any) associated with it is thrown away too, unexecuted.
+
+Just that much is almost enough to prevent problems, by throwing away
+*almost* all the weakref callbacks that could get triggered by gc. The
+problem remaining is that clearing a weakref with a callback decrefs the
+callback object, and the callback object may *itself* be weakly referenced,
+via another weakref with another callback. So the process of clearing
+weakrefs can trigger callbacks attached to other weakrefs, and those
+latter weakrefs may or may not be part of cyclic trash.
+
+So, to prevent any Python code from running while gc is invoking tp_clear()
+on all the objects in cyclic trash, it's not quite enough just to invoke
+tp_clear() on weakrefs with callbacks first. Instead the weakref module
+grew a new private function (_PyWeakref_ClearRef) that does only part of
+tp_clear(): it removes the weakref from the weakly-referenced object's list
+of weakrefs, but does not decref the callback object. So calling
+_PyWeakref_ClearRef(wr) ensures that wr's callback object will never
+trigger, and (unlike weakref's tp_clear()) also prevents any callback
+associated *with* wr's callback object from triggering.
+
+Then we can call tp_clear on all the cyclic objects and never trigger
+Python code.
+
+After we do that, the callback objects still need to be decref'ed. Callbacks
+(if any) *on* the callback objects that were also part of cyclic trash won't
+get invoked, because we cleared all trash weakrefs with callbacks at the
+start. Callbacks on the callback objects that were not part of cyclic trash
+acted as external roots to everything reachable from them, so nothing
+reachable from them was part of cyclic trash, so gc didn't do any damage to
+objects reachable from them, and it's safe to call them at the end of gc.
+
+An alternative would have been to treat objects with callbacks like objects
+with __del__ methods, refusing to collect them, appending them to gc.garbage
+instead. That would have been much easier. Jim Fulton gave a strong
+argument against that (on Python-Dev):
+
+ There's a big difference between __del__ and weakref callbacks.
+ The __del__ method is "internal" to a design. When you design a
+ class with a del method, you know you have to avoid including the
+ class in cycles.
+
+ Now, suppose you have a design that makes has no __del__ methods but
+ that does use cyclic data structures. You reason about the design,
+ run tests, and convince yourself you don't have a leak.
+
+ Now, suppose some external code creates a weakref to one of your
+ objects. All of a sudden, you start leaking. You can look at your
+ code all you want and you won't find a reason for the leak.
+
+IOW, a class designer can out-think __del__ problems, but has no control
+over who creates weakrefs to his classes or class instances. The class
+user has little chance either of predicting when the weakrefs he creates
+may end up in cycles.
+
+Callbacks on weakref callbacks are executed in an arbitrary order, and
+that's not good (a primary reason not to collect cycles with objects with
+__del__ methods is to avoid running finalizers in an arbitrary order).
+However, a weakref callback on a weakref callback has got to be rare.
+It's possible to do such a thing, so gc has to be robust against it, but
+I doubt anyone has done it outside the test case I wrote for it.
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index e6aabe4..7976b40 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -396,13 +396,17 @@
return 0;
}
-/* Move the objects in unreachable with __del__ methods into finalizers.
- * The objects remaining in unreachable do not have __del__ methods, and
- * gc_refs remains GC_TENTATIVELY_UNREACHABLE for them. The objects
- * moved into finalizers have gc_refs changed to GC_REACHABLE.
+/* Move the objects in unreachable with __del__ methods into finalizers,
+ * and weakrefs with callbacks into wr_callbacks.
+ * The objects remaining in unreachable do not have __del__ methods, and are
+ * not weakrefs with callbacks.
+ * The objects moved have gc_refs changed to GC_REACHABLE; the objects
+ * remaining in unreachable are left at GC_TENTATIVELY_UNREACHABLE.
*/
static void
-move_finalizers(PyGC_Head *unreachable, PyGC_Head *finalizers)
+move_troublemakers(PyGC_Head *unreachable,
+ PyGC_Head *finalizers,
+ PyGC_Head *wr_callbacks)
{
PyGC_Head *gc = unreachable->gc.gc_next;
@@ -417,6 +421,12 @@
gc_list_append(gc, finalizers);
gc->gc.gc_refs = GC_REACHABLE;
}
+ else if (PyWeakref_Check(op) &&
+ ((PyWeakReference *)op)->wr_callback) {
+ gc_list_remove(gc);
+ gc_list_append(gc, wr_callbacks);
+ gc->gc.gc_refs = GC_REACHABLE;
+ }
gc = next;
}
}
@@ -453,6 +463,93 @@
}
}
+/* Clear all trash weakrefs with callbacks. This clears weakrefs first,
+ * which has the happy result of disabling the callbacks without executing
+ * them. A nasty technical complication: a weakref callback can itself be
+ * the target of a weakref, in which case decrefing the callback can cause
+ * another callback to trigger. But we can't allow arbitrary Python code to
+ * get executed at this point (the callback on the callback may try to muck
+ * with other cyclic trash we're trying to collect, even resurrecting it
+ * while we're in the middle of doing tp_clear() on the trash).
+ *
+ * The private _PyWeakref_ClearRef() function exists so that we can clear
+ * the reference in a weakref without triggering a callback on the callback.
+ *
+ * We have to save the callback objects and decref them later. But we can't
+ * allocate new memory to save them (if we can't get new memory, we're dead).
+ * So we grab a new reference on the clear'ed weakref, which prevents the
+ * rest of gc from reclaiming it. _PyWeakref_ClearRef() leaves the
+ * weakref's wr_callback member intact.
+ *
+ * In the end, then, wr_callbacks consists of cleared weakrefs that are
+ * immune from collection. Near the end of gc, after collecting all the
+ * cyclic trash, we call release_weakrefs(). That releases our references
+ * to the cleared weakrefs, which in turn may trigger callbacks on their
+ * callbacks.
+ */
+static void
+clear_weakrefs(PyGC_Head *wr_callbacks)
+{
+ PyGC_Head *gc = wr_callbacks->gc.gc_next;
+
+ for (; gc != wr_callbacks; gc = gc->gc.gc_next) {
+ PyObject *op = FROM_GC(gc);
+ PyWeakReference *wr;
+
+ assert(IS_REACHABLE(op));
+ assert(PyWeakref_Check(op));
+ wr = (PyWeakReference *)op;
+ assert(wr->wr_callback != NULL);
+ Py_INCREF(op);
+ _PyWeakref_ClearRef(wr);
+ }
+}
+
+/* Called near the end of gc. This gives up the references we own to
+ * cleared weakrefs, allowing them to get collected, and in turn decref'ing
+ * their callbacks.
+ *
+ * If a callback object is itself the target of a weakref callback,
+ * decref'ing the callback object may trigger that other callback. If
+ * that other callback was part of the cyclic trash in this generation,
+ * that won't happen, since we cleared *all* trash-weakref callbacks near
+ * the start of gc. If that other callback was not part of the cyclic trash
+ * in this generation, then it acted like an external root to this round
+ * of gc, so all the objects reachable from that callback are still alive.
+ *
+ * Giving up the references to the weakref objects will probably make
+ * them go away too. However, if a weakref is reachable from finalizers,
+ * it won't go away. We move it to the old generation then. Since a
+ * weakref object doesn't have a finalizer, that's the right thing to do (it
+ * doesn't belong in gc.garbage).
+ *
+ * We return the number of weakref objects freed (those not appended to old).
+ */
+static int
+release_weakrefs(PyGC_Head *wr_callbacks, PyGC_Head *old)
+{
+ int num_freed = 0;
+
+ while (! gc_list_is_empty(wr_callbacks)) {
+ PyGC_Head *gc = wr_callbacks->gc.gc_next;
+ PyObject *op = FROM_GC(gc);
+ PyWeakReference *wr = (PyWeakReference *)op;
+
+ assert(IS_REACHABLE(op));
+ assert(PyWeakref_Check(op));
+ assert(wr->wr_callback != NULL);
+ Py_DECREF(op);
+ if (wr_callbacks->gc.gc_next == gc) {
+ /* object is still alive -- move it */
+ gc_list_remove(gc);
+ gc_list_append(gc, old);
+ }
+ else
+ ++num_freed;
+ }
+ return num_freed;
+}
+
static void
debug_instance(char *msg, PyInstanceObject *inst)
{
@@ -554,8 +651,9 @@
long n = 0; /* # unreachable objects that couldn't be collected */
PyGC_Head *young; /* the generation we are examining */
PyGC_Head *old; /* next older generation */
- PyGC_Head unreachable;
- PyGC_Head finalizers;
+ PyGC_Head unreachable; /* non-problematic unreachable trash */
+ PyGC_Head finalizers; /* objects with, & reachable from, __del__ */
+ PyGC_Head wr_callbacks; /* weakrefs with callbacks */
PyGC_Head *gc;
if (delstr == NULL) {
@@ -616,20 +714,33 @@
/* All objects in unreachable are trash, but objects reachable from
* finalizers can't safely be deleted. Python programmers should take
* care not to create such things. For Python, finalizers means
- * instance objects with __del__ methods.
+ * instance objects with __del__ methods. Weakrefs with callbacks
+ * can call arbitrary Python code, so those are special-cased too.
*
- * Move unreachable objects with finalizers into a different list.
+ * Move unreachable objects with finalizers, and weakrefs with
+ * callbacks, into different lists.
*/
gc_list_init(&finalizers);
- move_finalizers(&unreachable, &finalizers);
+ gc_list_init(&wr_callbacks);
+ move_troublemakers(&unreachable, &finalizers, &wr_callbacks);
+ /* Clear the trash weakrefs with callbacks. This prevents their
+ * callbacks from getting invoked (when a weakref goes away, so does
+ * its callback).
+ * We do this even if the weakrefs are reachable from finalizers.
+ * If we didn't, breaking cycles in unreachable later could trigger
+ * deallocation of objects in finalizers, which could in turn
+ * cause callbacks to trigger. This may not be ideal behavior.
+ */
+ clear_weakrefs(&wr_callbacks);
/* finalizers contains the unreachable objects with a finalizer;
- * unreachable objects reachable only *from* those are also
- * uncollectable, and we move those into the finalizers list too.
+ * unreachable objects reachable *from* those are also uncollectable,
+ * and we move those into the finalizers list too.
*/
move_finalizer_reachable(&finalizers);
/* Collect statistics on collectable objects found and print
- * debugging information. */
+ * debugging information.
+ */
for (gc = unreachable.gc.gc_next; gc != &unreachable;
gc = gc->gc.gc_next) {
m++;
@@ -643,6 +754,11 @@
*/
delete_garbage(&unreachable, old);
+ /* Now that we're done analyzing stuff and breaking cycles, let
+ * delayed weakref callbacks run.
+ */
+ m += release_weakrefs(&wr_callbacks, old);
+
/* Collect statistics on uncollectable objects found and print
* debugging information. */
for (gc = finalizers.gc.gc_next;