Gitiles
Code Review Sign In
gerrit-public.fairphone.software / platform / external / python / cpython3 / 7ade6da8660a640651a7e37f1b77b2721045942b / . / Demo / metaclasses / index.html
blob: 52d52e9fd01e520c0fc8ec75b9ed70a663c5094b [file] [log] [blame]
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]1<HTML>
2
3<HEAD>
Guido van Rossum7ade6da1997-09-11 22:54:49 +0000[diff] [blame^]4<TITLE>Metaclasses in Python 1.5</TITLE>
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]5</HEAD>
6
7<BODY BGCOLOR="FFFFFF">
8
Guido van Rossum7ade6da1997-09-11 22:54:49 +0000[diff] [blame^]9<H1>Metaclasses in Python 1.5</H1>
10<H2>(A.k.a. The Killer Joke :-)</H2>
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]11
Guido van Rossum7ade6da1997-09-11 22:54:49 +0000[diff] [blame^]12<P><b>Note: this document describes a feature only released in <A
13HREF="../../1.5a3/">Python 1.5a3</A>.</b>
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]14
15<P>In previous Python releases (and still in 1.5), there is something
16called the ``Don Beaudry hook'', after its inventor and champion.
17This allows C extensions to provide alternate class behavior, thereby
18allowing the Python class syntax to be used to define other class-like
19entities. Don Beaudry has used this in his infamous <A
20HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> package; Jim
21Fulton has used it in his <A
22HREF="http://www.digicool.com/papers/ExtensionClass.html">Extension
23Classes</A> package. (It has also been referred to as the ``Don
Guido van Rossum7b877a91997-09-08 02:20:57 +0000[diff] [blame]24Beaudry <i>hack</i>,'' but that's a misnomer. There's nothing hackish
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]25about it -- in fact, it is rather elegant and deep, even though
26there's something dark to it.)
27
28<P>Documentation of the Don Beaudry hook has purposefully been kept
29minimal, since it is a feature of incredible power, and is easily
30abused. Basically, it checks whether the <b>type of the base
31class</b> is callable, and if so, it is called to create the new
32class.
33
34<P>Note the two indirection levels. Take a simple example:
35
36<PRE>
37class B:
38 pass
39
40class C(B):
41 pass
42</PRE>
43
44Take a look at the second class definition, and try to fathom ``the
45type of the base class is callable.''
46
47<P>(Types are not classes, by the way. See questions 4.2, 4.19 and in
48particular 6.22 in the <A
49HREF="http://grail.cnri.reston.va.us/cgi-bin/faqw.py" >Python FAQ</A>
50for more on this topic.)
51
52<P>
53
54<UL>
55
56<LI>The <b>base class</b> is B; this one's easy.<P>
57
58<LI>Since B is a class, its type is ``class''; so the <b>type of the
59base class</b> is the type ``class''. This is also known as
60types.ClassType, assuming the standard module <code>types</code> has
61been imported.<P>
62
63<LI>Now is the type ``class'' <b>callable</b>? No, because types (in
64core Python) are never callable. Classes are callable (calling a
65class creates a new instance) but types aren't.<P>
66
67</UL>
68
69<P>So our conclusion is that in our example, the type of the base
70class (of C) is not callable. So the Don Beaudry hook does not apply,
71and the default class creation mechanism is used (which is also used
72when there is no base class). In fact, the Don Beaudry hook never
73applies when using only core Python, since the type of a core object
74is never callable.
75
76<P>So what do Don and Jim do in order to use Don's hook? Write an
77extension that defines at least two new Python object types. The
78first would be the type for ``class-like'' objects usable as a base
79class, to trigger Don's hook. This type must be made callable.
80That's why we need a second type. Whether an object is callable
81depends on its type. So whether a type object is callable depends on
82<i>its</i> type, which is a <i>meta-type</i>. (In core Python there
83is only one meta-type, the type ``type'' (types.TypeType), which is
84the type of all type objects, even itself.) A new meta-type must
85be defined that makes the type of the class-like objects callable.
86(Normally, a third type would also be needed, the new ``instance''
87type, but this is not an absolute requirement -- the new class type
88could return an object of some existing type when invoked to create an
89instance.)
90
91<P>Still confused? Here's a simple device due to Don himself to
92explain metaclasses. Take a simple class definition; assume B is a
93special class that triggers Don's hook:
94
95<PRE>
96class C(B):
97 a = 1
98 b = 2
99</PRE>
100
101This can be though of as equivalent to:
102
103<PRE>
104C = type(B)('C', (B,), {'a': 1, 'b': 2})
105</PRE>
106
107If that's too dense for you, here's the same thing written out using
108temporary variables:
109
110<PRE>
111creator = type(B) # The type of the base class
112name = 'C' # The name of the new class
113bases = (B,) # A tuple containing the base class(es)
114namespace = {'a': 1, 'b': 2} # The namespace of the class statement
115C = creator(name, bases, namespace)
116</PRE>
117
118This is analogous to what happens without the Don Beaudry hook, except
119that in that case the creator function is set to the default class
120creator.
121
122<P>In either case, the creator is called with three arguments. The
123first one, <i>name</i>, is the name of the new class (as given at the
124top of the class statement). The <i>bases</i> argument is a tuple of
125base classes (a singleton tuple if there's only one base class, like
126the example). Finally, <i>namespace</i> is a dictionary containing
127the local variables collected during execution of the class statement.
128
129<P>Note that the contents of the namespace dictionary is simply
130whatever names were defined in the class statement. A little-known
131fact is that when Python executes a class statement, it enters a new
132local namespace, and all assignments and function definitions take
133place in this namespace. Thus, after executing the following class
134statement:
135
136<PRE>
137class C:
138 a = 1
139 def f(s): pass
140</PRE>
141
142the class namespace's contents would be {'a': 1, 'f': &lt;function f
143...&gt;}.
144
Guido van Rossum7ade6da1997-09-11 22:54:49 +0000[diff] [blame^]145<P>But enough already about writing Python metaclasses in C; read the
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]146documentation of <A
147HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> or <A
148HREF="http://www.digicool.com/papers/ExtensionClass.html" >Extension
149Classes</A> for more information.
150
151<H2>Writing Metaclasses in Python</H2>
152
153<P>In Python 1.5, the requirement to write a C extension in order to
Guido van Rossum7ade6da1997-09-11 22:54:49 +0000[diff] [blame^]154write metaclasses has been dropped (though you can still do
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]155it, of course). In addition to the check ``is the type of the base
156class callable,'' there's a check ``does the base class have a
157__class__ attribute.'' If so, it is assumed that the __class__
158attribute refers to a class.
159
160<P>Let's repeat our simple example from above:
161
162<PRE>
163class C(B):
164 a = 1
165 b = 2
166</PRE>
167
168Assuming B has a __class__ attribute, this translates into:
169
170<PRE>
171C = B.__class__('C', (B,), {'a': 1, 'b': 2})
172</PRE>
173
174This is exactly the same as before except that instead of type(B),
175B.__class__ is invoked. If you have read <A HREF=
176"http://grail.cnri.reston.va.us/cgi-bin/faqw.py?req=show&file=faq06.022.htp"
177>FAQ question 6.22</A> you will understand that while there is a big
178technical difference between type(B) and B.__class__, they play the
179same role at different abstraction levels. And perhaps at some point
180in the future they will really be the same thing (at which point you
181would be able to derive subclasses from built-in types).
182
Guido van Rossum7b877a91997-09-08 02:20:57 +0000[diff] [blame]183<P>At this point it may be worth mentioning that C.__class__ is the
184same object as B.__class__, i.e., C's metaclass is the same as B's
185metaclass. In other words, subclassing an existing class creates a
186new (meta)inststance of the base class's metaclass.
187
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]188<P>Going back to the example, the class B.__class__ is instantiated,
189passing its constructor the same three arguments that are passed to
Guido van Rossum7ade6da1997-09-11 22:54:49 +0000[diff] [blame^]190the default class constructor or to an extension's metaclass:
191<i>name</i>, <i>bases</i>, and <i>namespace</i>.
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]192
193<P>It is easy to be confused by what exactly happens when using a
194metaclass, because we lose the absolute distinction between classes
195and instances: a class is an instance of a metaclass (a
196``metainstance''), but technically (i.e. in the eyes of the python
197runtime system), the metaclass is just a class, and the metainstance
198is just an instance. At the end of the class statement, the metaclass
199whose metainstance is used as a base class is instantiated, yielding a
200second metainstance (of the same metaclass). This metainstance is
201then used as a (normal, non-meta) class; instantiation of the class
202means calling the metainstance, and this will return a real instance.
203And what class is that an instance of? Conceptually, it is of course
204an instance of our metainstance; but in most cases the Python runtime
205system will see it as an instance of a a helper class used by the
206metaclass to implement its (non-meta) instances...
207
208<P>Hopefully an example will make things clearer. Let's presume we
209have a metaclass MetaClass1. It's helper class (for non-meta
210instances) is callled HelperClass1. We now (manually) instantiate
211MetaClass1 once to get an empty special base class:
212
213<PRE>
214BaseClass1 = MetaClass1("BaseClass1", (), {})
215</PRE>
216
217We can now use BaseClass1 as a base class in a class statement:
218
219<PRE>
220class MySpecialClass(BaseClass1):
221 i = 1
222 def f(s): pass
223</PRE>
224
225At this point, MySpecialClass is defined; it is a metainstance of
226MetaClass1 just like BaseClass1, and in fact the expression
227``BaseClass1.__class__ == MySpecialClass.__class__ == MetaClass1''
228yields true.
229
230<P>We are now ready to create instances of MySpecialClass. Let's
231assume that no constructor arguments are required:
232
233<PRE>
234x = MySpecialClass()
Guido van Rossum7b877a91997-09-08 02:20:57 +0000[diff] [blame]235y = MySpecialClass()
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]236print x.__class__, y.__class__
237</PRE>
238
239The print statement shows that x and y are instances of HelperClass1.
240How did this happen? MySpecialClass is an instance of MetaClass1
241(``meta'' is irrelevant here); when an instance is called, its
242__call__ method is invoked, and presumably the __call__ method defined
243by MetaClass1 returns an instance of HelperClass1.
244
Guido van Rossum7ade6da1997-09-11 22:54:49 +0000[diff] [blame^]245<P>Now let's see how we could use metaclasses -- what can we do
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]246with metaclasses that we can't easily do without them? Here's one
247idea: a metaclass could automatically insert trace calls for all
248method calls. Let's first develop a simplified example, without
249support for inheritance or other ``advanced'' Python features (we'll
250add those later).
251
252<PRE>
253import types
254
255class Tracing:
256 def __init__(self, name, bases, namespace):
257 """Create a new class."""
258 self.__name__ = name
259 self.__bases__ = bases
260 self.__namespace__ = namespace
261 def __call__(self):
262 """Create a new instance."""
263 return Instance(self)
264
265class Instance:
266 def __init__(self, klass):
267 self.__klass__ = klass
268 def __getattr__(self, name):
269 try:
270 value = self.__klass__.__namespace__[name]
271 except KeyError:
272 raise AttributeError, name
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]273 if type(value) is not types.FunctionType:
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]274 return value
275 return BoundMethod(value, self)
276
277class BoundMethod:
278 def __init__(self, function, instance):
279 self.function = function
280 self.instance = instance
281 def __call__(self, *args):
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]282 print "calling", self.function, "for", self.instance, "with", args
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]283 return apply(self.function, (self.instance,) + args)
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]284
285Trace = Tracing('Trace', (), {})
286
287class MyTracedClass(Trace):
288 def method1(self, a):
289 self.a = a
290 def method2(self):
291 return self.a
292
293aninstance = MyTracedClass()
294
295aninstance.method1(10)
296
297print "the answer is %d" % aninstance.method2()
298</PRE>
299
300Confused already? The intention is to read this from top down. The
301Tracing class is the metaclass we're defining. Its structure is
302really simple.
303
304<P>
305
306<UL>
307
308<LI>The __init__ method is invoked when a new Tracing instance is
309created, e.g. the definition of class MyTracedClass later in the
310example. It simply saves the class name, base classes and namespace
311as instance variables.<P>
312
313<LI>The __call__ method is invoked when a Tracing instance is called,
314e.g. the creation of aninstance later in the example. It returns an
315instance of the class Instance, which is defined next.<P>
316
317</UL>
318
319<P>The class Instance is the class used for all instances of classes
320built using the Tracing metaclass, e.g. aninstance. It has two
321methods:
322
323<P>
324
325<UL>
326
327<LI>The __init__ method is invoked from the Tracing.__call__ method
328above to initialize a new instance. It saves the class reference as
329an instance variable. It uses a funny name because the user's
330instance variables (e.g. self.a later in the example) live in the same
331namespace.<P>
332
333<LI>The __getattr__ method is invoked whenever the user code
334references an attribute of the instance that is not an instance
335variable (nor a class variable; but except for __init__ and
336__getattr__ there are no class variables). It will be called, for
337example, when aninstance.method1 is referenced in the example, with
338self set to aninstance and name set to the string "method1".<P>
339
340</UL>
341
342<P>The __getattr__ method looks the name up in the __namespace__
343dictionary. If it isn't found, it raises an AttributeError exception.
344(In a more realistic example, it would first have to look through the
345base classes as well.) If it is found, there are two possibilities:
346it's either a function or it isn't. If it's not a function, it is
347assumed to be a class variable, and its value is returned. If it's a
348function, we have to ``wrap'' it in instance of yet another helper
349class, BoundMethod.
350
351<P>The BoundMethod class is needed to implement a familiar feature:
352when a method is defined, it has an initial argument, self, which is
353automatically bound to the relevant instance when it is called. For
354example, aninstance.method1(10) is equivalent to method1(aninstance,
35510). In the example if this call, first a temporary BoundMethod
356instance is created with the following constructor call: temp =
357BoundMethod(method1, aninstance); then this instance is called as
358temp(10). After the call, the temporary instance is discarded.
359
360<P>
361
362<UL>
363
364<LI>The __init__ method is invoked for the constructor call
365BoundMethod(method1, aninstance). It simply saves away its
366arguments.<P>
367
368<LI>The __call__ method is invoked when the bound method instance is
369called, as in temp(10). It needs to call method1(aninstance, 10).
370However, even though self.function is now method1 and self.instance is
371aninstance, it can't call self.function(self.instance, args) directly,
372because it should work regardless of the number of arguments passed.
373(For simplicity, support for keyword arguments has been omitted.)<P>
374
375</UL>
376
377<P>In order to be able to support arbitrary argument lists, the
378__call__ method first constructs a new argument tuple. Conveniently,
379because of the notation *args in __call__'s own argument list, the
380arguments to __call__ (except for self) are placed in the tuple args.
381To construct the desired argument list, we concatenate a singleton
382tuple containing the instance with the args tuple: (self.instance,) +
383args. (Note the trailing comma used to construct the singleton
384tuple.) In our example, the resulting argument tuple is (aninstance,
38510).
386
387<P>The intrinsic function apply() takes a function and an argument
388tuple and calls the function for it. In our example, we are calling
389apply(method1, (aninstance, 10)) which is equivalent to calling
390method(aninstance, 10).
391
392<P>From here on, things should come together quite easily. The output
393of the example code is something like this:
394
395<PRE>
396calling <function method1 at ae8d8> for <Instance instance at 95ab0> with (10,)
397calling <function method2 at ae900> for <Instance instance at 95ab0> with ()
398the answer is 10
399</PRE>
400
401<P>That was about the shortest meaningful example that I could come up
402with. A real tracing metaclass (for example, <A
403HREF="#Trace">Trace.py</A> discussed below) needs to be more
404complicated in two dimensions.
405
406<P>First, it needs to support more advanced Python features such as
407class variables, inheritance, __init__ methods, and keyword arguments.
408
409<P>Second, it needs to provide a more flexible way to handle the
410actual tracing information; perhaps it should be possible to write
411your own tracing function that gets called, perhaps it should be
412possible to enable and disable tracing on a per-class or per-instance
413basis, and perhaps a filter so that only interesting calls are traced;
414it should also be able to trace the return value of the call (or the
415exception it raised if an error occurs). Even the Trace.py example
416doesn't support all these features yet.
417
418<P>
419
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]420<HR>
421
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]422<H1>Real-life Examples</H1>
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]423
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]424<P>Have a look at some very preliminary examples that I coded up to
Guido van Rossum7ade6da1997-09-11 22:54:49 +0000[diff] [blame^]425teach myself how to write metaclasses:
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]426
427<DL>
428
429<DT><A HREF="Enum.py">Enum.py</A>
430
431<DD>This (ab)uses the class syntax as an elegant way to define
432enumerated types. The resulting classes are never instantiated --
433rather, their class attributes are the enumerated values. For
434example:
435
436<PRE>
437class Color(Enum):
438 red = 1
439 green = 2
440 blue = 3
441print Color.red
442</PRE>
443
444will print the string ``Color.red'', while ``Color.red==1'' is true,
445and ``Color.red + 1'' raise a TypeError exception.
446
447<P>
448
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]449<DT><A NAME=Trace></A><A HREF="Trace.py">Trace.py</A>
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]450
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]451<DD>The resulting classes work much like standard
452classes, but by setting a special class or instance attribute
453__trace_output__ to point to a file, all calls to the class's methods
454are traced. It was a bit of a struggle to get this right. This
455should probably redone using the generic metaclass below.
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]456
457<P>
458
459<DT><A HREF="Meta.py">Meta.py</A>
460
461<DD>A generic metaclass. This is an attempt at finding out how much
462standard class behavior can be mimicked by a metaclass. The
463preliminary answer appears to be that everything's fine as long as the
464class (or its clients) don't look at the instance's __class__
465attribute, nor at the class's __dict__ attribute. The use of
466__getattr__ internally makes the classic implementation of __getattr__
467hooks tough; we provide a similar hook _getattr_ instead.
468(__setattr__ and __delattr__ are not affected.)
469(XXX Hm. Could detect presence of __getattr__ and rename it.)
470
471<P>
472
473<DT><A HREF="Eiffel.py">Eiffel.py</A>
Guido van Rossum7b877a91997-09-08 02:20:57 +0000[diff] [blame]474
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]475<DD>Uses the above generic metaclass to implement Eiffel style
476pre-conditions and post-conditions.
477
478<P>
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]479
480<DT><A HREF="Synch.py">Synch.py</A>
481
482<DD>Uses the above generic metaclass to implement synchronized
483methods.
484
485<P>
486
Guido van Rossum7b877a91997-09-08 02:20:57 +0000[diff] [blame]487<DT><A HREF="Simple.py">Simple.py</A>
488
489<DD>The example module used above.
490
491<P>
492
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]493</DL>
494
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]495<P>A pattern seems to be emerging: almost all these uses of
496metaclasses (except for Enum, which is probably more cute than useful)
497mostly work by placing wrappers around method calls. An obvious
498problem with that is that it's not easy to combine the features of
499different metaclasses, while this would actually be quite useful: for
500example, I wouldn't mind getting a trace from the test run of the
501Synch module, and it would be interesting to add preconditions to it
502as well. This needs more research. Perhaps a metaclass could be
503provided that allows stackable wrappers...
504
Guido van Rossum7b877a91997-09-08 02:20:57 +0000[diff] [blame]505<HR>
506
507<H2>Things You Could Do With Metaclasses</H2>
508
509<P>There are lots of things you could do with metaclasses. Most of
510these can also be done with creative use of __getattr__, but
511metaclasses make it easier to modify the attribute lookup behavior of
512classes. Here's a partial list.
513
514<P>
515
516<UL>
517
518<LI>Enforce different inheritance semantics, e.g. automatically call
519base class methods when a derived class overrides<P>
520
521<LI>Implement class methods (e.g. if the first argument is not named
522'self')<P>
523
524<LI>Implement that each instance is initialized with <b>copies</b> of
525all class variables<P>
526
527<LI>Implement a different way to store instance variables (e.g. in a
528list kept outside the the instance but indexed by the instance's id())<P>
529
530<LI>Automatically wrap or trap all or certain methods
531
532<UL>
533
534<LI>for tracing
535
536<LI>for precondition and postcondition checking
537
538<LI>for synchronized methods
539
540<LI>for automatic value caching
541
542</UL>
543<P>
544
545<LI>When an attribute is a parameterless function, call it on
546reference (to mimic it being an instance variable); same on assignment<P>
547
548<LI>Instrumentation: see how many times various attributes are used<P>
549
550<LI>Different semantics for __setattr__ and __getattr__ (e.g. disable
551them when they are being used recursively)<P>
552
553<LI>Abuse class syntax for other things<P>
554
555<LI>Experiment with automatic type checking<P>
556
557<LI>Delegation (or acquisition)<P>
558
559<LI>Dynamic inheritance patterns<P>
560
561<LI>Automatic caching of methods<P>
562
563</UL>
564
565<P>
566
567<HR>
568
569<H4>Credits</H4>
570
571<P>Many thanks to David Ascher and Donald Beaudry for their comments
572on earlier draft of this paper. Also thanks to Matt Conway and Tommy
573Burnette for putting a seed for the idea of metaclasses in my
574mind, nearly three years ago, even though at the time my response was
575``you can do that with __getattr__ hooks...'' :-)
576
577<P>
578
579<HR>
580
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]581</BODY>
582
583</HTML>