Demo/metaclasses/index.html - platform/external/python/cpython3 - Gitiles

 <HTML>

 <HEAD>
 <TITLE>Metaclasses in Python 1.5</TITLE>
 </HEAD>

 <BODY BGCOLOR="FFFFFF">

 <H1>Metaclasses in Python 1.5</H1>
 <H2>(A.k.a. The Killer Joke :-)</H2>

 <HR>

 (<i>Postscript:</i> reading this essay is probably not the best way to
 understand the metaclass hook described here.  See a <A
 HREF="meta-vladimir.txt">message posted by Vladimir Marangozov</A>
 which may give a gentler introduction to the matter.  You may also
 want to search Deja News for messages with "metaclass" in the subject
 posted to comp.lang.python in July and August 1998.)

 <HR>

 <P>In previous Python releases (and still in 1.5), there is something
 called the ``Don Beaudry hook'', after its inventor and champion.
 This allows C extensions to provide alternate class behavior, thereby
 allowing the Python class syntax to be used to define other class-like
 entities.  Don Beaudry has used this in his infamous <A
 HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> package; Jim
 Fulton has used it in his <A
 HREF="http://www.digicool.com/releases/ExtensionClass/">Extension
 Classes</A> package.  (It has also been referred to as the ``Don
 Beaudry <i>hack</i>,'' but that's a misnomer.  There's nothing hackish
 about it -- in fact, it is rather elegant and deep, even though
 there's something dark to it.)

 <P>(On first reading, you may want to skip directly to the examples in
 the section "Writing Metaclasses in Python" below, unless you want
 your head to explode.)

 <P>

 <HR>

 <P>Documentation of the Don Beaudry hook has purposefully been kept
 minimal, since it is a feature of incredible power, and is easily
 abused.  Basically, it checks whether the <b>type of the base
 class</b> is callable, and if so, it is called to create the new
 class.

 <P>Note the two indirection levels.  Take a simple example:

 <PRE>
 class B:
     pass

 class C(B):
     pass
 </PRE>

 Take a look at the second class definition, and try to fathom ``the
 type of the base class is callable.''

 <P>(Types are not classes, by the way.  See questions 4.2, 4.19 and in
 particular 6.22 in the <A
 HREF="http://www.python.org/cgi-bin/faqw.py" >Python FAQ</A>
 for more on this topic.)

 <P>

 <UL>

 <LI>The <b>base class</b> is B; this one's easy.<P>

 <LI>Since B is a class, its type is ``class''; so the <b>type of the
 base class</b> is the type ``class''.  This is also known as
 types.ClassType, assuming the standard module <code>types</code> has
 been imported.<P>

 <LI>Now is the type ``class'' <b>callable</b>?  No, because types (in
 core Python) are never callable.  Classes are callable (calling a
 class creates a new instance) but types aren't.<P>

 </UL>

 <P>So our conclusion is that in our example, the type of the base
 class (of C) is not callable.  So the Don Beaudry hook does not apply,
 and the default class creation mechanism is used (which is also used
 when there is no base class).  In fact, the Don Beaudry hook never
 applies when using only core Python, since the type of a core object
 is never callable.

 <P>So what do Don and Jim do in order to use Don's hook?  Write an
 extension that defines at least two new Python object types.  The
 first would be the type for ``class-like'' objects usable as a base
 class, to trigger Don's hook.  This type must be made callable.
 That's why we need a second type.  Whether an object is callable
 depends on its type.  So whether a type object is callable depends on
 <i>its</i> type, which is a <i>meta-type</i>.  (In core Python there
 is only one meta-type, the type ``type'' (types.TypeType), which is
 the type of all type objects, even itself.)  A new meta-type must
 be defined that makes the type of the class-like objects callable.
 (Normally, a third type would also be needed, the new ``instance''
 type, but this is not an absolute requirement -- the new class type
 could return an object of some existing type when invoked to create an
 instance.)

 <P>Still confused?  Here's a simple device due to Don himself to
 explain metaclasses.  Take a simple class definition; assume B is a
 special class that triggers Don's hook:

 <PRE>
 class C(B):
     a = 1
     b = 2
 </PRE>

 This can be though of as equivalent to:

 <PRE>
 C = type(B)('C', (B,), {'a': 1, 'b': 2})
 </PRE>

 If that's too dense for you, here's the same thing written out using
 temporary variables:

 <PRE>
 creator = type(B)               # The type of the base class
 name = 'C'                      # The name of the new class
 bases = (B,)                    # A tuple containing the base class(es)
 namespace = {'a': 1, 'b': 2}    # The namespace of the class statement
 C = creator(name, bases, namespace)
 </PRE>

 This is analogous to what happens without the Don Beaudry hook, except
 that in that case the creator function is set to the default class
 creator.

 <P>In either case, the creator is called with three arguments.  The
 first one, <i>name</i>, is the name of the new class (as given at the
 top of the class statement).  The <i>bases</i> argument is a tuple of
 base classes (a singleton tuple if there's only one base class, like
 the example).  Finally, <i>namespace</i> is a dictionary containing
 the local variables collected during execution of the class statement.

 <P>Note that the contents of the namespace dictionary is simply
 whatever names were defined in the class statement.  A little-known
 fact is that when Python executes a class statement, it enters a new
 local namespace, and all assignments and function definitions take
 place in this namespace.  Thus, after executing the following class
 statement:

 <PRE>
 class C:
     a = 1
     def f(s): pass
 </PRE>

 the class namespace's contents would be {'a': 1, 'f': &lt;function f
 ...&gt;}.

 <P>But enough already about writing Python metaclasses in C; read the
 documentation of <A
 HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> or <A
 HREF="http://www.digicool.com/papers/ExtensionClass.html" >Extension
 Classes</A> for more information.

 <P>

 <HR>

 <H2>Writing Metaclasses in Python</H2>

 <P>In Python 1.5, the requirement to write a C extension in order to
 write metaclasses has been dropped (though you can still do
 it, of course).  In addition to the check ``is the type of the base
 class callable,'' there's a check ``does the base class have a
 __class__ attribute.''  If so, it is assumed that the __class__
 attribute refers to a class.

 <P>Let's repeat our simple example from above:

 <PRE>
 class C(B):
     a = 1
     b = 2
 </PRE>

 Assuming B has a __class__ attribute, this translates into:

 <PRE>
 C = B.__class__('C', (B,), {'a': 1, 'b': 2})
 </PRE>

 This is exactly the same as before except that instead of type(B),
 B.__class__ is invoked.  If you have read <A HREF=
 "http://www.python.org/cgi-bin/faqw.py?req=show&file=faq06.022.htp"
 >FAQ question 6.22</A> you will understand that while there is a big
 technical difference between type(B) and B.__class__, they play the
 same role at different abstraction levels.  And perhaps at some point
 in the future they will really be the same thing (at which point you
 would be able to derive subclasses from built-in types).

 <P>At this point it may be worth mentioning that C.__class__ is the
 same object as B.__class__, i.e., C's metaclass is the same as B's
 metaclass.  In other words, subclassing an existing class creates a
 new (meta)inststance of the base class's metaclass.

 <P>Going back to the example, the class B.__class__ is instantiated,
 passing its constructor the same three arguments that are passed to
 the default class constructor or to an extension's metaclass:
 <i>name</i>, <i>bases</i>, and <i>namespace</i>.

 <P>It is easy to be confused by what exactly happens when using a
 metaclass, because we lose the absolute distinction between classes
 and instances: a class is an instance of a metaclass (a
 ``metainstance''), but technically (i.e. in the eyes of the python
 runtime system), the metaclass is just a class, and the metainstance
 is just an instance.  At the end of the class statement, the metaclass
 whose metainstance is used as a base class is instantiated, yielding a
 second metainstance (of the same metaclass).  This metainstance is
 then used as a (normal, non-meta) class; instantiation of the class
 means calling the metainstance, and this will return a real instance.
 And what class is that an instance of?  Conceptually, it is of course
 an instance of our metainstance; but in most cases the Python runtime
 system will see it as an instance of a a helper class used by the
 metaclass to implement its (non-meta) instances...

 <P>Hopefully an example will make things clearer.  Let's presume we
 have a metaclass MetaClass1.  It's helper class (for non-meta
 instances) is callled HelperClass1.  We now (manually) instantiate
 MetaClass1 once to get an empty special base class:

 <PRE>
 BaseClass1 = MetaClass1("BaseClass1", (), {})
 </PRE>

 We can now use BaseClass1 as a base class in a class statement:

 <PRE>
 class MySpecialClass(BaseClass1):
     i = 1
     def f(s): pass
 </PRE>

 At this point, MySpecialClass is defined; it is a metainstance of
 MetaClass1 just like BaseClass1, and in fact the expression
 ``BaseClass1.__class__ == MySpecialClass.__class__ == MetaClass1''
 yields true.

 <P>We are now ready to create instances of MySpecialClass.  Let's
 assume that no constructor arguments are required:

 <PRE>
 x = MySpecialClass()
 y = MySpecialClass()
 print x.__class__, y.__class__
 </PRE>

 The print statement shows that x and y are instances of HelperClass1.
 How did this happen?  MySpecialClass is an instance of MetaClass1
 (``meta'' is irrelevant here); when an instance is called, its
 __call__ method is invoked, and presumably the __call__ method defined
 by MetaClass1 returns an instance of HelperClass1.

 <P>Now let's see how we could use metaclasses -- what can we do
 with metaclasses that we can't easily do without them?  Here's one
 idea: a metaclass could automatically insert trace calls for all
 method calls.  Let's first develop a simplified example, without
 support for inheritance or other ``advanced'' Python features (we'll
 add those later).

 <PRE>
 import types

 class Tracing:
     def __init__(self, name, bases, namespace):
         """Create a new class."""
         self.__name__ = name
         self.__bases__ = bases
         self.__namespace__ = namespace
     def __call__(self):
         """Create a new instance."""
         return Instance(self)

 class Instance:
     def __init__(self, klass):
         self.__klass__ = klass
     def __getattr__(self, name):
         try:
             value = self.__klass__.__namespace__[name]
         except KeyError:
             raise AttributeError, name
         if type(value) is not types.FunctionType:
             return value
         return BoundMethod(value, self)

 class BoundMethod:
     def __init__(self, function, instance):
         self.function = function
         self.instance = instance
     def __call__(self, *args):
         print "calling", self.function, "for", self.instance, "with", args
         return apply(self.function, (self.instance,) + args)

 Trace = Tracing('Trace', (), {})

 class MyTracedClass(Trace):
     def method1(self, a):
         self.a = a
     def method2(self):
         return self.a

 aninstance = MyTracedClass()

 aninstance.method1(10)

 print "the answer is %d" % aninstance.method2()
 </PRE>

 Confused already?  The intention is to read this from top down.  The
 Tracing class is the metaclass we're defining.  Its structure is
 really simple.

 <P>

 <UL>

 <LI>The __init__ method is invoked when a new Tracing instance is
 created, e.g. the definition of class MyTracedClass later in the
 example.  It simply saves the class name, base classes and namespace
 as instance variables.<P>

 <LI>The __call__ method is invoked when a Tracing instance is called,
 e.g. the creation of aninstance later in the example.  It returns an
 instance of the class Instance, which is defined next.<P>

 </UL>

 <P>The class Instance is the class used for all instances of classes
 built using the Tracing metaclass, e.g. aninstance.  It has two
 methods:

 <P>

 <UL>

 <LI>The __init__ method is invoked from the Tracing.__call__ method
 above to initialize a new instance.  It saves the class reference as
 an instance variable.  It uses a funny name because the user's
 instance variables (e.g. self.a later in the example) live in the same
 namespace.<P>

 <LI>The __getattr__ method is invoked whenever the user code
 references an attribute of the instance that is not an instance
 variable (nor a class variable; but except for __init__ and
 __getattr__ there are no class variables).  It will be called, for
 example, when aninstance.method1 is referenced in the example, with
 self set to aninstance and name set to the string "method1".<P>

 </UL>

 <P>The __getattr__ method looks the name up in the __namespace__
 dictionary.  If it isn't found, it raises an AttributeError exception.
 (In a more realistic example, it would first have to look through the
 base classes as well.)  If it is found, there are two possibilities:
 it's either a function or it isn't.  If it's not a function, it is
 assumed to be a class variable, and its value is returned.  If it's a
 function, we have to ``wrap'' it in instance of yet another helper
 class, BoundMethod.

 <P>The BoundMethod class is needed to implement a familiar feature:
 when a method is defined, it has an initial argument, self, which is
 automatically bound to the relevant instance when it is called.  For
 example, aninstance.method1(10) is equivalent to method1(aninstance,
 10).  In the example if this call, first a temporary BoundMethod
 instance is created with the following constructor call: temp =
 BoundMethod(method1, aninstance); then this instance is called as
 temp(10).  After the call, the temporary instance is discarded.

 <P>

 <UL>

 <LI>The __init__ method is invoked for the constructor call
 BoundMethod(method1, aninstance).  It simply saves away its
 arguments.<P>

 <LI>The __call__ method is invoked when the bound method instance is
 called, as in temp(10).  It needs to call method1(aninstance, 10).
 However, even though self.function is now method1 and self.instance is
 aninstance, it can't call self.function(self.instance, args) directly,
 because it should work regardless of the number of arguments passed.
 (For simplicity, support for keyword arguments has been omitted.)<P>

 </UL>

 <P>In order to be able to support arbitrary argument lists, the
 __call__ method first constructs a new argument tuple.  Conveniently,
 because of the notation *args in __call__'s own argument list, the
 arguments to __call__ (except for self) are placed in the tuple args.
 To construct the desired argument list, we concatenate a singleton
 tuple containing the instance with the args tuple: (self.instance,) +
 args.  (Note the trailing comma used to construct the singleton
 tuple.)  In our example, the resulting argument tuple is (aninstance,
 10).

 <P>The intrinsic function apply() takes a function and an argument
 tuple and calls the function for it.  In our example, we are calling
 apply(method1, (aninstance, 10)) which is equivalent to calling
 method(aninstance, 10).

 <P>From here on, things should come together quite easily.  The output
 of the example code is something like this:

 <PRE>
 calling &lt;function method1 at ae8d8&gt; for &lt;Instance instance at 95ab0&gt; with (10,)
 calling &lt;function method2 at ae900&gt; for &lt;Instance instance at 95ab0&gt; with ()
 the answer is 10
 </PRE>

 <P>That was about the shortest meaningful example that I could come up
 with.  A real tracing metaclass (for example, <A
 HREF="#Trace">Trace.py</A> discussed below) needs to be more
 complicated in two dimensions.

 <P>First, it needs to support more advanced Python features such as
 class variables, inheritance, __init__ methods, and keyword arguments.

 <P>Second, it needs to provide a more flexible way to handle the
 actual tracing information; perhaps it should be possible to write
 your own tracing function that gets called, perhaps it should be
 possible to enable and disable tracing on a per-class or per-instance
 basis, and perhaps a filter so that only interesting calls are traced;
 it should also be able to trace the return value of the call (or the
 exception it raised if an error occurs).  Even the Trace.py example
 doesn't support all these features yet.

 <P>

 <HR>

 <H1>Real-life Examples</H1>

 <P>Have a look at some very preliminary examples that I coded up to
 teach myself how to write metaclasses:

 <DL>

 <DT><A HREF="Enum.py">Enum.py</A>

 <DD>This (ab)uses the class syntax as an elegant way to define
 enumerated types.  The resulting classes are never instantiated --
 rather, their class attributes are the enumerated values.  For
 example:

 <PRE>
 class Color(Enum):
     red = 1
     green = 2
     blue = 3
 print Color.red
 </PRE>

 will print the string ``Color.red'', while ``Color.red==1'' is true,
 and ``Color.red + 1'' raise a TypeError exception.

 <P>

 <DT><A NAME=Trace></A><A HREF="Trace.py">Trace.py</A>

 <DD>The resulting classes work much like standard
 classes, but by setting a special class or instance attribute
 __trace_output__ to point to a file, all calls to the class's methods
 are traced.  It was a bit of a struggle to get this right.  This
 should probably redone using the generic metaclass below.

 <P>

 <DT><A HREF="Meta.py">Meta.py</A>

 <DD>A generic metaclass.  This is an attempt at finding out how much
 standard class behavior can be mimicked by a metaclass.  The
 preliminary answer appears to be that everything's fine as long as the
 class (or its clients) don't look at the instance's __class__
 attribute, nor at the class's __dict__ attribute.  The use of
 __getattr__ internally makes the classic implementation of __getattr__
 hooks tough; we provide a similar hook _getattr_ instead.
 (__setattr__ and __delattr__ are not affected.)
 (XXX Hm.  Could detect presence of __getattr__ and rename it.)

 <P>

 <DT><A HREF="Eiffel.py">Eiffel.py</A>

 <DD>Uses the above generic metaclass to implement Eiffel style
 pre-conditions and post-conditions.

 <P>

 <DT><A HREF="Synch.py">Synch.py</A>

 <DD>Uses the above generic metaclass to implement synchronized
 methods.

 <P>

 <DT><A HREF="Simple.py">Simple.py</A>

 <DD>The example module used above.

 <P>

 </DL>

 <P>A pattern seems to be emerging: almost all these uses of
 metaclasses (except for Enum, which is probably more cute than useful)
 mostly work by placing wrappers around method calls.  An obvious
 problem with that is that it's not easy to combine the features of
 different metaclasses, while this would actually be quite useful: for
 example, I wouldn't mind getting a trace from the test run of the
 Synch module, and it would be interesting to add preconditions to it
 as well.  This needs more research.  Perhaps a metaclass could be
 provided that allows stackable wrappers...

 <P>

 <HR>

 <H2>Things You Could Do With Metaclasses</H2>

 <P>There are lots of things you could do with metaclasses.  Most of
 these can also be done with creative use of __getattr__, but
 metaclasses make it easier to modify the attribute lookup behavior of
 classes.  Here's a partial list.

 <P>

 <UL>

 <LI>Enforce different inheritance semantics, e.g. automatically call
 base class methods when a derived class overrides<P>

 <LI>Implement class methods (e.g. if the first argument is not named
 'self')<P>

 <LI>Implement that each instance is initialized with <b>copies</b> of
 all class variables<P>

 <LI>Implement a different way to store instance variables (e.g. in a
 list kept outside the the instance but indexed by the instance's id())<P>

 <LI>Automatically wrap or trap all or certain methods

 <UL>

 <LI>for tracing

 <LI>for precondition and postcondition checking

 <LI>for synchronized methods

 <LI>for automatic value caching

 </UL>
 <P>

 <LI>When an attribute is a parameterless function, call it on
 reference (to mimic it being an instance variable); same on assignment<P>

 <LI>Instrumentation: see how many times various attributes are used<P>

 <LI>Different semantics for __setattr__ and __getattr__ (e.g.  disable
 them when they are being used recursively)<P>

 <LI>Abuse class syntax for other things<P>

 <LI>Experiment with automatic type checking<P>

 <LI>Delegation (or acquisition)<P>

 <LI>Dynamic inheritance patterns<P>

 <LI>Automatic caching of methods<P>

 </UL>

 <P>

 <HR>

 <H4>Credits</H4>

 <P>Many thanks to David Ascher and Donald Beaudry for their comments
 on earlier draft of this paper.  Also thanks to Matt Conway and Tommy
 Burnette for putting a seed for the idea of metaclasses in my
 mind, nearly three years ago, even though at the time my response was
 ``you can do that with __getattr__ hooks...'' :-)

 <P>

 <HR>

 </BODY>

 </HTML>
	<HTML>

	<HEAD>
	<TITLE>Metaclasses in Python 1.5</TITLE>
	</HEAD>

	<BODY BGCOLOR="FFFFFF">

	<H1>Metaclasses in Python 1.5</H1>
	<H2>(A.k.a. The Killer Joke :-)</H2>

	<HR>

	(<i>Postscript:</i> reading this essay is probably not the best way to
	understand the metaclass hook described here. See a <A
	HREF="meta-vladimir.txt">message posted by Vladimir Marangozov</A>
	which may give a gentler introduction to the matter. You may also
	want to search Deja News for messages with "metaclass" in the subject
	posted to comp.lang.python in July and August 1998.)

	<HR>

	<P>In previous Python releases (and still in 1.5), there is something
	called the ``Don Beaudry hook'', after its inventor and champion.
	This allows C extensions to provide alternate class behavior, thereby
	allowing the Python class syntax to be used to define other class-like
	entities. Don Beaudry has used this in his infamous <A
	HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> package; Jim
	Fulton has used it in his <A
	HREF="http://www.digicool.com/releases/ExtensionClass/">Extension
	Classes</A> package. (It has also been referred to as the ``Don
	Beaudry <i>hack</i>,'' but that's a misnomer. There's nothing hackish
	about it -- in fact, it is rather elegant and deep, even though
	there's something dark to it.)

	<P>(On first reading, you may want to skip directly to the examples in
	the section "Writing Metaclasses in Python" below, unless you want
	your head to explode.)

	<P>

	<HR>

	<P>Documentation of the Don Beaudry hook has purposefully been kept
	minimal, since it is a feature of incredible power, and is easily
	abused. Basically, it checks whether the <b>type of the base
	class</b> is callable, and if so, it is called to create the new
	class.

	<P>Note the two indirection levels. Take a simple example:

	<PRE>
	class B:
	pass

	class C(B):
	pass
	</PRE>

	Take a look at the second class definition, and try to fathom ``the
	type of the base class is callable.''

	<P>(Types are not classes, by the way. See questions 4.2, 4.19 and in
	particular 6.22 in the <A
	HREF="http://www.python.org/cgi-bin/faqw.py" >Python FAQ</A>
	for more on this topic.)

	<P>

	<UL>

	<LI>The <b>base class</b> is B; this one's easy.<P>

	<LI>Since B is a class, its type is ``class''; so the <b>type of the
	base class</b> is the type ``class''. This is also known as
	types.ClassType, assuming the standard module <code>types</code> has
	been imported.<P>

	<LI>Now is the type ``class'' <b>callable</b>? No, because types (in
	core Python) are never callable. Classes are callable (calling a
	class creates a new instance) but types aren't.<P>

	</UL>

	<P>So our conclusion is that in our example, the type of the base
	class (of C) is not callable. So the Don Beaudry hook does not apply,
	and the default class creation mechanism is used (which is also used
	when there is no base class). In fact, the Don Beaudry hook never
	applies when using only core Python, since the type of a core object
	is never callable.

	<P>So what do Don and Jim do in order to use Don's hook? Write an
	extension that defines at least two new Python object types. The
	first would be the type for ``class-like'' objects usable as a base
	class, to trigger Don's hook. This type must be made callable.
	That's why we need a second type. Whether an object is callable
	depends on its type. So whether a type object is callable depends on
	<i>its</i> type, which is a <i>meta-type</i>. (In core Python there
	is only one meta-type, the type ``type'' (types.TypeType), which is
	the type of all type objects, even itself.) A new meta-type must
	be defined that makes the type of the class-like objects callable.
	(Normally, a third type would also be needed, the new ``instance''
	type, but this is not an absolute requirement -- the new class type
	could return an object of some existing type when invoked to create an
	instance.)

	<P>Still confused? Here's a simple device due to Don himself to
	explain metaclasses. Take a simple class definition; assume B is a
	special class that triggers Don's hook:

	<PRE>
	class C(B):
	a = 1
	b = 2
	</PRE>

	This can be though of as equivalent to:

	<PRE>
	C = type(B)('C', (B,), {'a': 1, 'b': 2})
	</PRE>

	If that's too dense for you, here's the same thing written out using
	temporary variables:

	<PRE>
	creator = type(B) # The type of the base class
	name = 'C' # The name of the new class
	bases = (B,) # A tuple containing the base class(es)
	namespace = {'a': 1, 'b': 2} # The namespace of the class statement
	C = creator(name, bases, namespace)
	</PRE>

	This is analogous to what happens without the Don Beaudry hook, except
	that in that case the creator function is set to the default class
	creator.

	<P>In either case, the creator is called with three arguments. The
	first one, <i>name</i>, is the name of the new class (as given at the
	top of the class statement). The <i>bases</i> argument is a tuple of
	base classes (a singleton tuple if there's only one base class, like
	the example). Finally, <i>namespace</i> is a dictionary containing
	the local variables collected during execution of the class statement.

	<P>Note that the contents of the namespace dictionary is simply
	whatever names were defined in the class statement. A little-known
	fact is that when Python executes a class statement, it enters a new
	local namespace, and all assignments and function definitions take
	place in this namespace. Thus, after executing the following class
	statement:

	<PRE>
	class C:
	a = 1
	def f(s): pass
	</PRE>

	the class namespace's contents would be {'a': 1, 'f': <function f
	...>}.

	<P>But enough already about writing Python metaclasses in C; read the
	documentation of <A
	HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> or <A
	HREF="http://www.digicool.com/papers/ExtensionClass.html" >Extension
	Classes</A> for more information.

	<P>

	<HR>

	<H2>Writing Metaclasses in Python</H2>

	<P>In Python 1.5, the requirement to write a C extension in order to
	write metaclasses has been dropped (though you can still do
	it, of course). In addition to the check ``is the type of the base
	class callable,'' there's a check ``does the base class have a
	__class__ attribute.'' If so, it is assumed that the __class__
	attribute refers to a class.

	<P>Let's repeat our simple example from above:

	<PRE>
	class C(B):
	a = 1
	b = 2
	</PRE>

	Assuming B has a __class__ attribute, this translates into:

	<PRE>
	C = B.__class__('C', (B,), {'a': 1, 'b': 2})
	</PRE>

	This is exactly the same as before except that instead of type(B),
	B.__class__ is invoked. If you have read <A HREF=
	"http://www.python.org/cgi-bin/faqw.py?req=show&file=faq06.022.htp"
	>FAQ question 6.22</A> you will understand that while there is a big
	technical difference between type(B) and B.__class__, they play the
	same role at different abstraction levels. And perhaps at some point
	in the future they will really be the same thing (at which point you
	would be able to derive subclasses from built-in types).

	<P>At this point it may be worth mentioning that C.__class__ is the
	same object as B.__class__, i.e., C's metaclass is the same as B's
	metaclass. In other words, subclassing an existing class creates a
	new (meta)inststance of the base class's metaclass.

	<P>Going back to the example, the class B.__class__ is instantiated,
	passing its constructor the same three arguments that are passed to
	the default class constructor or to an extension's metaclass:
	<i>name</i>, <i>bases</i>, and <i>namespace</i>.

	<P>It is easy to be confused by what exactly happens when using a
	metaclass, because we lose the absolute distinction between classes
	and instances: a class is an instance of a metaclass (a
	``metainstance''), but technically (i.e. in the eyes of the python
	runtime system), the metaclass is just a class, and the metainstance
	is just an instance. At the end of the class statement, the metaclass
	whose metainstance is used as a base class is instantiated, yielding a
	second metainstance (of the same metaclass). This metainstance is
	then used as a (normal, non-meta) class; instantiation of the class
	means calling the metainstance, and this will return a real instance.
	And what class is that an instance of? Conceptually, it is of course
	an instance of our metainstance; but in most cases the Python runtime
	system will see it as an instance of a a helper class used by the
	metaclass to implement its (non-meta) instances...

	<P>Hopefully an example will make things clearer. Let's presume we
	have a metaclass MetaClass1. It's helper class (for non-meta
	instances) is callled HelperClass1. We now (manually) instantiate
	MetaClass1 once to get an empty special base class:

	<PRE>
	BaseClass1 = MetaClass1("BaseClass1", (), {})
	</PRE>

	We can now use BaseClass1 as a base class in a class statement:

	<PRE>
	class MySpecialClass(BaseClass1):
	i = 1
	def f(s): pass
	</PRE>

	At this point, MySpecialClass is defined; it is a metainstance of
	MetaClass1 just like BaseClass1, and in fact the expression
	``BaseClass1.__class__ == MySpecialClass.__class__ == MetaClass1''
	yields true.

	<P>We are now ready to create instances of MySpecialClass. Let's
	assume that no constructor arguments are required:

	<PRE>
	x = MySpecialClass()
	y = MySpecialClass()
	print x.__class__, y.__class__
	</PRE>

	The print statement shows that x and y are instances of HelperClass1.
	How did this happen? MySpecialClass is an instance of MetaClass1
	(``meta'' is irrelevant here); when an instance is called, its
	__call__ method is invoked, and presumably the __call__ method defined
	by MetaClass1 returns an instance of HelperClass1.

	<P>Now let's see how we could use metaclasses -- what can we do
	with metaclasses that we can't easily do without them? Here's one
	idea: a metaclass could automatically insert trace calls for all
	method calls. Let's first develop a simplified example, without
	support for inheritance or other ``advanced'' Python features (we'll
	add those later).

	<PRE>
	import types

	class Tracing:
	def __init__(self, name, bases, namespace):
	"""Create a new class."""
	self.__name__ = name
	self.__bases__ = bases
	self.__namespace__ = namespace
	def __call__(self):
	"""Create a new instance."""
	return Instance(self)

	class Instance:
	def __init__(self, klass):
	self.__klass__ = klass
	def __getattr__(self, name):
	try:
	value = self.__klass__.__namespace__[name]
	except KeyError:
	raise AttributeError, name
	if type(value) is not types.FunctionType:
	return value
	return BoundMethod(value, self)

	class BoundMethod:
	def __init__(self, function, instance):
	self.function = function
	self.instance = instance
	def __call__(self, *args):
	print "calling", self.function, "for", self.instance, "with", args
	return apply(self.function, (self.instance,) + args)

	Trace = Tracing('Trace', (), {})

	class MyTracedClass(Trace):
	def method1(self, a):
	self.a = a
	def method2(self):
	return self.a

	aninstance = MyTracedClass()

	aninstance.method1(10)

	print "the answer is %d" % aninstance.method2()
	</PRE>

	Confused already? The intention is to read this from top down. The
	Tracing class is the metaclass we're defining. Its structure is
	really simple.

	<P>

	<UL>

	<LI>The __init__ method is invoked when a new Tracing instance is
	created, e.g. the definition of class MyTracedClass later in the
	example. It simply saves the class name, base classes and namespace
	as instance variables.<P>

	<LI>The __call__ method is invoked when a Tracing instance is called,
	e.g. the creation of aninstance later in the example. It returns an
	instance of the class Instance, which is defined next.<P>

	</UL>

	<P>The class Instance is the class used for all instances of classes
	built using the Tracing metaclass, e.g. aninstance. It has two
	methods:

	<P>

	<UL>

	<LI>The __init__ method is invoked from the Tracing.__call__ method
	above to initialize a new instance. It saves the class reference as
	an instance variable. It uses a funny name because the user's
	instance variables (e.g. self.a later in the example) live in the same
	namespace.<P>

	<LI>The __getattr__ method is invoked whenever the user code
	references an attribute of the instance that is not an instance
	variable (nor a class variable; but except for __init__ and
	__getattr__ there are no class variables). It will be called, for
	example, when aninstance.method1 is referenced in the example, with
	self set to aninstance and name set to the string "method1".<P>

	</UL>

	<P>The __getattr__ method looks the name up in the __namespace__
	dictionary. If it isn't found, it raises an AttributeError exception.
	(In a more realistic example, it would first have to look through the
	base classes as well.) If it is found, there are two possibilities:
	it's either a function or it isn't. If it's not a function, it is
	assumed to be a class variable, and its value is returned. If it's a
	function, we have to ``wrap'' it in instance of yet another helper
	class, BoundMethod.

	<P>The BoundMethod class is needed to implement a familiar feature:
	when a method is defined, it has an initial argument, self, which is
	automatically bound to the relevant instance when it is called. For
	example, aninstance.method1(10) is equivalent to method1(aninstance,
	10). In the example if this call, first a temporary BoundMethod
	instance is created with the following constructor call: temp =
	BoundMethod(method1, aninstance); then this instance is called as
	temp(10). After the call, the temporary instance is discarded.

	<P>

	<UL>

	<LI>The __init__ method is invoked for the constructor call
	BoundMethod(method1, aninstance). It simply saves away its
	arguments.<P>

	<LI>The __call__ method is invoked when the bound method instance is
	called, as in temp(10). It needs to call method1(aninstance, 10).
	However, even though self.function is now method1 and self.instance is
	aninstance, it can't call self.function(self.instance, args) directly,
	because it should work regardless of the number of arguments passed.
	(For simplicity, support for keyword arguments has been omitted.)<P>

	</UL>

	<P>In order to be able to support arbitrary argument lists, the
	__call__ method first constructs a new argument tuple. Conveniently,
	because of the notation *args in __call__'s own argument list, the
	arguments to __call__ (except for self) are placed in the tuple args.
	To construct the desired argument list, we concatenate a singleton
	tuple containing the instance with the args tuple: (self.instance,) +
	args. (Note the trailing comma used to construct the singleton
	tuple.) In our example, the resulting argument tuple is (aninstance,
	10).

	<P>The intrinsic function apply() takes a function and an argument
	tuple and calls the function for it. In our example, we are calling
	apply(method1, (aninstance, 10)) which is equivalent to calling
	method(aninstance, 10).

	<P>From here on, things should come together quite easily. The output
	of the example code is something like this:

	<PRE>
	calling <function method1 at ae8d8> for <Instance instance at 95ab0> with (10,)
	calling <function method2 at ae900> for <Instance instance at 95ab0> with ()
	the answer is 10
	</PRE>

	<P>That was about the shortest meaningful example that I could come up
	with. A real tracing metaclass (for example, <A
	HREF="#Trace">Trace.py</A> discussed below) needs to be more
	complicated in two dimensions.

	<P>First, it needs to support more advanced Python features such as
	class variables, inheritance, __init__ methods, and keyword arguments.

	<P>Second, it needs to provide a more flexible way to handle the
	actual tracing information; perhaps it should be possible to write
	your own tracing function that gets called, perhaps it should be
	possible to enable and disable tracing on a per-class or per-instance
	basis, and perhaps a filter so that only interesting calls are traced;
	it should also be able to trace the return value of the call (or the
	exception it raised if an error occurs). Even the Trace.py example
	doesn't support all these features yet.

	<P>

	<HR>

	<H1>Real-life Examples</H1>

	<P>Have a look at some very preliminary examples that I coded up to
	teach myself how to write metaclasses:

	<DL>

	<DT><A HREF="Enum.py">Enum.py</A>

	<DD>This (ab)uses the class syntax as an elegant way to define
	enumerated types. The resulting classes are never instantiated --
	rather, their class attributes are the enumerated values. For
	example:

	<PRE>
	class Color(Enum):
	red = 1
	green = 2
	blue = 3
	print Color.red
	</PRE>

	will print the string ``Color.red'', while ``Color.red==1'' is true,
	and ``Color.red + 1'' raise a TypeError exception.

	<P>

	<DT><A NAME=Trace></A><A HREF="Trace.py">Trace.py</A>

	<DD>The resulting classes work much like standard
	classes, but by setting a special class or instance attribute
	__trace_output__ to point to a file, all calls to the class's methods
	are traced. It was a bit of a struggle to get this right. This
	should probably redone using the generic metaclass below.

	<P>

	<DT><A HREF="Meta.py">Meta.py</A>

	<DD>A generic metaclass. This is an attempt at finding out how much
	standard class behavior can be mimicked by a metaclass. The
	preliminary answer appears to be that everything's fine as long as the
	class (or its clients) don't look at the instance's __class__
	attribute, nor at the class's __dict__ attribute. The use of
	__getattr__ internally makes the classic implementation of __getattr__
	hooks tough; we provide a similar hook _getattr_ instead.
	(__setattr__ and __delattr__ are not affected.)
	(XXX Hm. Could detect presence of __getattr__ and rename it.)

	<P>

	<DT><A HREF="Eiffel.py">Eiffel.py</A>

	<DD>Uses the above generic metaclass to implement Eiffel style
	pre-conditions and post-conditions.

	<P>

	<DT><A HREF="Synch.py">Synch.py</A>

	<DD>Uses the above generic metaclass to implement synchronized
	methods.

	<P>

	<DT><A HREF="Simple.py">Simple.py</A>

	<DD>The example module used above.

	<P>

	</DL>

	<P>A pattern seems to be emerging: almost all these uses of
	metaclasses (except for Enum, which is probably more cute than useful)
	mostly work by placing wrappers around method calls. An obvious
	problem with that is that it's not easy to combine the features of
	different metaclasses, while this would actually be quite useful: for
	example, I wouldn't mind getting a trace from the test run of the
	Synch module, and it would be interesting to add preconditions to it
	as well. This needs more research. Perhaps a metaclass could be
	provided that allows stackable wrappers...

	<P>

	<HR>

	<H2>Things You Could Do With Metaclasses</H2>

	<P>There are lots of things you could do with metaclasses. Most of
	these can also be done with creative use of __getattr__, but
	metaclasses make it easier to modify the attribute lookup behavior of
	classes. Here's a partial list.

	<P>

	<UL>

	<LI>Enforce different inheritance semantics, e.g. automatically call
	base class methods when a derived class overrides<P>

	<LI>Implement class methods (e.g. if the first argument is not named
	'self')<P>

	<LI>Implement that each instance is initialized with <b>copies</b> of
	all class variables<P>

	<LI>Implement a different way to store instance variables (e.g. in a
	list kept outside the the instance but indexed by the instance's id())<P>

	<LI>Automatically wrap or trap all or certain methods

	<UL>

	<LI>for tracing

	<LI>for precondition and postcondition checking

	<LI>for synchronized methods

	<LI>for automatic value caching

	</UL>
	<P>

	<LI>When an attribute is a parameterless function, call it on
	reference (to mimic it being an instance variable); same on assignment<P>

	<LI>Instrumentation: see how many times various attributes are used<P>

	<LI>Different semantics for __setattr__ and __getattr__ (e.g. disable
	them when they are being used recursively)<P>

	<LI>Abuse class syntax for other things<P>

	<LI>Experiment with automatic type checking<P>

	<LI>Delegation (or acquisition)<P>

	<LI>Dynamic inheritance patterns<P>

	<LI>Automatic caching of methods<P>

	</UL>

	<P>

	<HR>

	<H4>Credits</H4>

	<P>Many thanks to David Ascher and Donald Beaudry for their comments
	on earlier draft of this paper. Also thanks to Matt Conway and Tommy
	Burnette for putting a seed for the idea of metaclasses in my
	mind, nearly three years ago, even though at the time my response was
	``you can do that with __getattr__ hooks...'' :-)

	<P>

	<HR>

	</BODY>

	</HTML>